From dff9cc42bf4963b01fbe9a42f54bef0cb55d2112 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Mon, 17 Aug 2026 12:20:32 -0700 Subject: [PATCH 001/107] Add WebGPU EP extraction plans Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: df9d994f-d4ed-4e61-8104-57abb42ae205 --- .../node_plugin_migration_workstream.md | 150 ++++++++ ...boundary_and_web_integration_workstream.md | 143 +++++++ ...ion_and_repository_migration_workstream.md | 232 +++++++++++ .../ep_operator_conformance_design.md | 362 ++++++++++++++++++ ...st_ownership_and_conformance_workstream.md | 195 ++++++++++ .../webgpu_ep_extraction.md | 148 +++++++ 6 files changed, 1230 insertions(+) create mode 100644 docs/design/webgpu_ep_extraction/node_plugin_migration/node_plugin_migration_workstream.md create mode 100644 docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md create mode 100644 docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md create mode 100644 docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md create mode 100644 docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md create mode 100644 docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md diff --git a/docs/design/webgpu_ep_extraction/node_plugin_migration/node_plugin_migration_workstream.md b/docs/design/webgpu_ep_extraction/node_plugin_migration/node_plugin_migration_workstream.md new file mode 100644 index 0000000000000..737f2f009323a --- /dev/null +++ b/docs/design/webgpu_ep_extraction/node_plugin_migration/node_plugin_migration_workstream.md @@ -0,0 +1,150 @@ +# Workstream `node-migration`: Node Plugin Migration + +Status: Working plan + +[WebGPU EP extraction overview](../webgpu_ep_extraction.md) + +## Objective + +Replace the WebGPU EP currently bundled in `onnxruntime-node` with an explicitly consumable plugin while preserving a +supported migration path for existing Node WebGPU users. + +This is a required extraction workstream, not optional new consumer enablement. Built-in Node WebGPU support must not +be removed until the replacement package and loading model are tested on the agreed launch platforms. + +## Current state + +`onnxruntime-node` currently: + +- Includes WebGPU in several prebuilt platform binaries. +- Recognizes the `"webgpu"` execution provider through compile-time provider-specific code. +- Accepts WebGPU-specific provider options. +- Supports WebGPU buffer interoperability. +- Uses a singleton native `Ort::Env`; JavaScript callers do not create independent ORT environment instances. + +Removing the provider from these binaries changes an existing, although experimental, capability and requires an +explicit compatibility and package transition. + +## Desired end state + +- The Node ORT host does not compile, bundle, or depend on the WebGPU implementation. +- Node can register optional plugin EP libraries through a generic API. +- A separately installable WebGPU package supplies supported platform-specific plugin artifacts. +- Existing WebGPU provider options and buffer interoperability continue to work. +- Loading errors identify missing packages, incompatible versions, unsupported platforms, and late registration. +- The mechanism supports other plugin EPs without adding provider-specific Node binding code. +- Existing users have a documented package and code migration path. + +## Generic Node plugin loading + +The Node binding should expose an API to register a plugin library before creating sessions that use it. + +The current binding owns a singleton `Ort::Env`, so the initial API should register plugins with that singleton. +If Node later exposes user-created ORT environment instances, registration can be extended to those instances. + +The API should define: + +- Whether registration is explicit or may also be triggered by a package helper. +- Required ordering relative to ORT initialization and session creation. +- Library lifetime and cleanup across Node worker environments. +- Duplicate registration behavior. +- Error handling for incompatible plugin and ORT versions. +- How provider names and options become visible to session creation. +- How a package safely resolves its platform-specific native library. + +## WebGPU npm package + +The WebGPU repository should publish an npm package that: + +- Contains or installs the appropriate native WebGPU plugin artifact for each supported platform and architecture. +- Exposes a small helper that resolves and registers the artifact through the generic Node API. +- Does not require WebGPU-specific code in the Node ORT binding. +- Declares compatibility independently from the ORT package version. +- Produces clear unsupported-platform and compatibility errors. +- Follows the same signing, provenance, and release requirements as comparable ORT packages. + +The exact package name is open. + +## Core Node package transition + +Two primary package strategies remain under consideration: + +| Strategy | Advantages | Costs and risks | +| --- | --- | --- | +| Keep `onnxruntime-node` as the core host and remove bundled WebGPU in a major-version transition | Preserves the established package name and avoids maintaining two core packages | Existing WebGPU users must install and register a second package; capability removal requires prominent migration guidance | +| Introduce a core-only package such as `onnxruntime-node-core` | Allows the existing `onnxruntime-node` contract to remain stable during migration and makes the optional boundary explicit | Creates a new ecosystem package, duplicates support or requires a later consolidation, and may confuse which package applications should choose | + +An extended compatibility period may accompany either strategy, but indefinitely bundling WebGPU is not the target +state. The decision should account for the experimental status of current WebGPU support, semantic-versioning policy, +download size, other built-in EPs, and maintenance cost. + +## Compatibility requirements + +The migration must preserve: + +- Session creation using the `"webgpu"` provider name or a clearly documented replacement. +- Existing provider options. +- GPU-buffer input and output behavior on supported platforms. +- Node worker behavior and native library lifetime safety. +- Current supported platform coverage unless a reduction is explicitly approved. +- Clear detection of ORT/plugin version incompatibility. + +## Tests + +Required coverage includes: + +- Installation of the core Node package without WebGPU. +- Installation and registration of the WebGPU plugin package. +- Session creation and inference with CPU fallback disabled. +- Existing WebGPU provider options. +- GPU-buffer interoperability. +- Missing-plugin, unsupported-platform, duplicate-registration, and version-mismatch failures. +- Node worker initialization and cleanup. +- Upgrade tests for the selected package transition. + +## Parallel work packages + +1. **API design:** define generic plugin registration for the singleton Node ORT environment. +2. **Binding implementation:** load and register arbitrary plugin EP libraries. +3. **Package prototype:** package WebGPU native artifacts and registration helper. +4. **Compatibility validation:** preserve options, buffers, workers, and diagnostics. +5. **Package transition decision:** select naming, versioning, and deprecation policy. +6. **Release migration:** publish packages, documentation, and upgrade tests before removing bundled WebGPU. + +The API and package prototypes can proceed in parallel once the native plugin artifact contract is known. + +## Interfaces with other workstreams + +### Plugin boundary and Web/Wasm integration + +- Reuses ORT's generic dynamic plugin registration and compatibility behavior. +- Does not depend on the static WebAssembly registration path. + +### Provider isolation and repository migration + +- Consumes versioned native shared WebGPU plugin artifacts. +- Coordinates platform naming, signing, compatibility metadata, and release timing. + +### Test ownership and operator conformance + +- Supplies Node host and package integration tests. +- Reuses portable operator cases where practical to verify the loaded provider executes correctly. + +## Completion criteria + +- A package naming and compatibility strategy is approved. +- The Node binding registers plugin EPs without WebGPU-specific compile-time code. +- A separately installable WebGPU package exists for the agreed launch platforms. +- Current WebGPU options and buffer interop pass against the plugin. +- Package installation, worker, compatibility, and failure-mode tests are blocking. +- Existing users have migration documentation and a supported transition window. +- Bundled WebGPU is removed only after the replacement is released and validated. + +## Open questions + +- Should the core host remain `onnxruntime-node` or move to a name such as `onnxruntime-node-core`? +- What should the WebGPU npm package be named? +- Should plugin registration be an explicit application call, a package helper side effect, or both? +- Which Node platforms and architectures are required at first release? +- How long should any compatibility or deprecation period last? +- How should plugin loading behave across Node worker environments? diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md new file mode 100644 index 0000000000000..c437a559aba74 --- /dev/null +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md @@ -0,0 +1,143 @@ +# Workstream `plugin-boundary`: Plugin Boundary and Web/Wasm Integration + +Status: Working plan + +[WebGPU EP extraction overview](../webgpu_ep_extraction.md) + +## Objective + +Make the public plugin EP interface the only runtime boundary between ONNX Runtime (ORT) and the WebGPU EP. + +The same WebGPU provider implementation must work in two linkage modes: + +| Host | Linkage | Factory discovery | +| --- | --- | --- | +| Native ORT | Shared plugin library | Runtime symbol lookup | +| `onnxruntime-web` and other static hosts | Static library | Direct factory registration | + +This workstream owns the generic ORT infrastructure and host integration needed to make those modes equivalent. It +does not own WebGPU kernels, Dawn, shader tooling, or the provider's standalone build and release system. + +## Desired end state + +- WebGPU implements `OrtEpFactory` and `OrtEp` once. +- Dynamic and static linkage differ only in factory discovery, symbol visibility, and lifetime wiring. +- No WebGPU path reaches into `IExecutionProvider`, `OpKernel`, `Tensor`, `KernelRegistry`, or other private ORT + interfaces. +- `onnxruntime-web` registers the statically linked factory through a generic ORT facility. +- Browser objects cross a narrow, documented boundary with explicit ownership and lifetime rules. +- Native hosts can load WebGPU as an optional plugin without core ORT packages depending on it. +- The legacy direct/static WebGPU provider path is removed after parity is demonstrated. + +## Scope + +### Generic static plugin registration + +Add an ORT facility that accepts statically linked plugin factory entry points. It should reuse the existing dynamic +plugin path after library loading and symbol lookup. + +The design must address: + +- Unique internal entry-point names when multiple static plugins are linked. +- Factory, device, allocator, data-transfer, and process-global lifetimes. +- Registration timing relative to environment creation. +- Cleanup when there is no dynamic-library unload event. +- Reduced and extended-minimal builds. +- Dead-code elimination for statically linked providers. +- Diagnostics for incompatible or duplicate registrations. + +The facility must be generic and validated with at least one non-WebGPU test plugin where practical. + +### WebGPU plugin-path parity + +Compile the existing WebGPU adapter implementation as both a shared plugin and a static plugin library. Exercise the +same factory, device discovery, provider options, allocator, data transfer, graph assignment, and execution code in +both forms. + +Parity work includes: + +- Native shared-plugin execution. +- Native static registration as a focused test host where useful. +- Emscripten static registration. +- Provider option behavior. +- GPU tensor and buffer interoperability. +- Error and diagnostic behavior. +- Process and session lifetime behavior. + +The direct in-tree WebGPU provider path remains only as a temporary comparison baseline. + +### Plugin API gap closure + +Provider-isolation work will identify uses of private ORT interfaces. Each finding should be resolved by: + +1. An existing public `OrtApi` or `OrtEpApi` operation. +2. A public plugin EP API addition when a confirmed gap represents a stable runtime boundary useful beyond WebGPU. +3. A WebGPU-owned helper or replacement in the `provider-isolation` workstream. + +Stable API additions have a high compatibility cost. Convenience helpers and provider implementation details should +not be moved into ORT's public API merely to simplify extraction. + +Likely investigation areas include: + +- External-data loading in WebAssembly. +- Graph and model information needed during capability discovery or compilation. +- Device tensors and externally owned buffers. +- Reduced-operator configuration. +- Logging, threading, allocators, and data transfer. +- Environment and process-global initialization. + +### Browser/Wasm bridge + +Define the smallest stable interface needed to pass JavaScript-owned WebGPU objects between ORT Web and the provider. + +The ORT repository should retain generic Wasm module assembly, JavaScript package behavior, and ORT lifecycle wiring. +The provider repository should own WebGPU-specific behavior. The boundary must define: + +- `GPUDevice` and `GPUBuffer` representation. +- Ownership, reference, and destruction rules. +- Threading and async behavior. +- Device-loss propagation. +- Validation and error reporting. +- Compatibility with JavaScript and Emscripten changes. + +The bridge should not expose unrelated ORT private implementation details. + +## Parallel work packages + +1. **Static registration core:** implement and contract-test generic static factory registration. +2. **Emscripten prototype:** compile the adapter path statically and run a small model. +3. **Gap inventory triage:** convert private-dependency findings into public API or provider-owned actions. +4. **Browser bridge:** specify and prototype object and lifetime exchange. +5. **Parity and retirement:** run the existing suite through the plugin path and remove the legacy path. + +The first four packages can proceed largely in parallel. Legacy-path removal waits for their convergence. + +## Interfaces with other workstreams + +### Provider isolation and repository migration + +- The `provider-isolation` workstream supplies concrete private-dependency findings. +- This workstream supplies the public plugin EP API headers and static host-registration contract. +- The `provider-isolation` workstream produces the static and shared libraries consumed by parity tests. + +### Test ownership and operator conformance + +- The `test-conformance` workstream supplies blocking parity cases and fallback detection. +- This workstream provides dynamic and static registration hooks for conformance runners. +- Contract tests for generic plugin infrastructure remain in ORT. + +## Completion criteria + +- Static and dynamic WebGPU builds use the same provider implementation and public API boundary. +- A WebGPU model executes through static registration in `onnxruntime-web`. +- Existing plugin-path tests are blocking and detect fallback. +- All required private-runtime interactions have a documented public API or provider-owned replacement. +- Browser object ownership and lifecycle are documented and tested. +- Reduced WebAssembly builds retain required plugin infrastructure within accepted size and startup budgets. +- The direct `IExecutionProvider` WebGPU path is removed. + +## Open questions + +- Should static factories be registered before environment creation or through environment construction options? +- How should process-global plugin state be finalized in static builds? +- What is the stable representation of JavaScript-owned WebGPU objects at the C API boundary? diff --git a/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md b/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md new file mode 100644 index 0000000000000..5b0d92cc7b44d --- /dev/null +++ b/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md @@ -0,0 +1,232 @@ +# Workstream `provider-isolation`: Provider Isolation and Repository Migration + +Status: Working plan + +[WebGPU EP extraction overview](../webgpu_ep_extraction.md) + +## Objective + +Make the WebGPU EP an independently buildable and releasable component, first under an isolated staging root in the +ORT repository and then in a dedicated repository. + +The repository move should become a controlled copy of an already independent subtree rather than a large refactor +performed at the same time as the move. + +## Desired end state + +The WebGPU repository owns: + +- `OrtEpFactory` and `OrtEp` provider implementation. +- WebGPU kernels, contrib kernels, runtime, and device infrastructure. +- WebGPU-owned support code copied or replaced from ORT. +- Dawn selection, patches, and build configuration. +- WGSL templates, generators, and generated-source policy. +- WebGPU-specific unit, integration, browser, package, and regression tests. +- Shared and static provider build targets. +- Existing Python and NuGet plugin packaging and release pipelines. +- Version metadata, compatibility policy, CI, and release artifacts. + +ORT should consume versioned artifacts or source and should not remain an implementation or packaging repository for +the provider. + +## Isolation strategy + +Expand the existing `plugin-ep-webgpu/` directory into the in-tree staging root. It already owns Python and NuGet +plugin packaging, version metadata, and release documentation. Add provider source, build, dependency, and test +ownership until the directory mirrors the intended external repository: + +```text +plugin-ep-webgpu/ + cmake/ + include/ + src/ + ep/ + kernels/ + runtime/ + support/ + third_party/ + dawn/ + tools/ + wgsl/ + tests/ + unit/ + operators/ + integration/ + conformance/ + python/ + csharp/ + VERSION_NUMBER + MIN_ONNXRUNTIME_VERSION +``` + +All provider-owned build inputs should be reachable from that root without consulting an implicit list of files +elsewhere in the ORT tree. + +During isolation: + +- Preserve selected history in the new repository through a filtered-history import, such as `git filter-repo` or an + equivalent subtree export. Moving files into `plugin-ep-webgpu/` first makes the ownership boundary and subsequent + history extraction clearer. The import should exclude unrelated ORT history where practical. +- Avoid changing behavior merely to change ownership. +- Keep ORT integration shims outside the provider root. +- Make generated files and downloaded dependencies explicit build outputs or inputs. +- Support an ORT build override pointing at an adjacent WebGPU checkout. + +## Private dependency removal + +Create a machine-reviewable inventory of: + +- Private ORT headers included by provider sources. +- Private ORT libraries in provider link interfaces. +- Source files compiled into WebGPU targets from outside the provider root. +- Build variables and generated files supplied by ORT. +- Test-only dependencies on ORT internals. +- Platform and package scripts that assume an in-tree provider. + +Classify each dependency: + +| Resolution | Use when | +| --- | --- | +| Existing public API | The plugin SDK already expresses the required runtime interaction | +| New public plugin API | The operation is a stable, generally useful runtime boundary | +| WebGPU-owned copy | The code is implementation support and can evolve independently | +| WebGPU-specific replacement | Existing ORT code is unsuitable as a cross-repository dependency | +| ORT integration shim | The behavior adapts an ORT host or build to the external provider but is not part of provider behavior | + +Examples of ORT integration shims include registering a statically linked factory during ORT Web startup, translating +an ORT reduced-operator configuration into provider build input, selecting a pinned provider source archive, or +adapting JavaScript-owned browser objects to the public bridge. These remain in ORT because they describe how an ORT +host consumes the provider. + +Copied code must preserve license and provenance. Once copied, it becomes WebGPU-owned code and is not synchronized +with the ORT implementation. + +## Standalone build contract + +The isolated subtree should build against a declared ORT plugin SDK or installed ORT package without an ORT source +checkout. + +It should produce: + +- A native shared plugin library. +- A static plugin-API library for Emscripten and other static hosts. +- Provider metadata sufficient to diagnose ORT compatibility. +- Test executables or packages that consume public ORT interfaces. +- Development artifacts from the same commit as release artifacts. + +The build should support: + +- Pinned and overridable ORT SDK locations. +- Pinned Dawn and other third-party dependencies. +- Reduced-operator input from an ORT Web build. +- Platform-specific symbol visibility and export rules. +- Reproducible source archives. +- Adjacent-checkout development from ORT. + +## Repository foundation + +The external repository skeleton can be created before isolation is complete to validate: + +- Directory and CMake layout. +- Required checks and platform matrix. +- Dependency caching and Dawn build time. +- Version and compatibility metadata. +- Artifact naming and retention. +- Issue ownership and contribution policy. +- Release automation. + +The final code import should wait until a clean copy of the staging root builds and tests without undeclared ORT +source-tree dependencies. + +## Packaging migration + +Python and NuGet plugin packaging already exist. This workstream migrates their sources and release pipelines rather +than redesigning them. + +The packaging model is: + +- Core ORT packages do not depend on or bundle WebGPU. +- Users install the WebGPU plugin package separately. +- Plugin packages declare compatible ORT versions and fail clearly on incompatibility. +- Deprecated packages with built-in WebGPU are retired instead of being converted into plugin-dependent packages. +- `onnxruntime-web` consumes an immutable source archive or commit for static linkage. + +The Node workstream consumes the native shared plugin artifacts produced here. Node package naming, plugin +registration, and compatibility behavior are owned end-to-end by +[Node plugin migration](../node_plugin_migration/node_plugin_migration_workstream.md). + +## Test relocation + +The `test-conformance` workstream classifies existing tests. This workstream physically moves tests classified as +WebGPU-owned, including: + +- Kernel and shader generation tests. +- Dawn integration and backend tests. +- Device-limit and feature tests. +- Caching and provider-option tests. +- Browser interop tests for WebGPU-specific behavior. +- Package installation and artifact tests. +- Provider implementation regressions. + +Portable operator cases and generic plugin contract tests remain owned by ORT. Temporary legacy tests may remain in +ORT until equivalent external coverage is blocking. + +## Parallel work packages + +1. **Dependency inventory:** enumerate includes, libraries, generated inputs, and build assumptions. +2. **Staging-root design:** define layout, targets, SDK inputs, and integration shims. +3. **Support-code isolation:** copy or replace implementation helpers. +4. **Dependency ownership:** move Dawn, patches, and WGSL generation. +5. **Standalone build:** produce static and shared artifacts outside the ORT build graph. +6. **Repository and CI scaffold:** validate clean-checkout development and release jobs. +7. **Packaging migration:** move the existing Python and NuGet pipelines. +8. **Source transfer:** copy the proven staging root and switch ORT to pinned consumption. + +Inventory, repository scaffolding, packaging analysis, and test classification can begin immediately. Source transfer +depends on the standalone build. + +## Interfaces with other workstreams + +### Plugin boundary and Web/Wasm integration + +- Private-dependency findings may create plugin API work. +- This workstream consumes public plugin EP API headers and static registration contracts. +- Browser-specific ownership must be agreed before moving bridge code. + +### Test ownership and operator conformance + +- The `test-conformance` workstream decides test ownership and minimum regression gates. +- This workstream supplies external test targets and CI environments. +- The provider profile, exclusions, and WebGPU-specific tests move with the provider. + +### Node plugin migration + +- This workstream supplies versioned native shared plugin artifacts. +- The Node workstream owns npm layout, loading, package naming, and user migration. + +## Completion criteria + +- All WebGPU-owned build inputs live under the isolated staging root. +- Shared and static libraries build from a clean copy without private ORT libraries or headers. +- Dawn and WGSL generation are provider-owned and reproducible. +- WebGPU-specific tests run from the isolated tree and remain blocking. +- Python and NuGet plugin packages are produced and tested from the isolated tree. +- The external repository CI builds, tests, and packages a clean checkout. +- ORT can consume a pinned external source artifact and can override it with an adjacent checkout. +- Native ORT packages remain WebGPU-independent. + +## Versioning, provenance, and remaining questions + +The WebGPU EP version is independent from the ORT version. Compatibility metadata declares the minimum and tested ORT +versions instead of coupling release numbers. + +Package signing, provenance, and release controls should meet the same requirements as comparable ORT core packages. + +ORT should consume WebGPU source using the standard mechanism used for comparable third-party source dependencies. +The dependency inventory should compare existing ORT mechanisms before selecting the exact implementation. + +Remaining questions: + +- Which copied ORT helpers need independent namespaces or API cleanup before transfer? +- How should reduced-operator configuration be represented as an external provider input? +- Which platforms and architectures are required for the first independent release? diff --git a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md new file mode 100644 index 0000000000000..6c2a6ffc1c852 --- /dev/null +++ b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md @@ -0,0 +1,362 @@ +# Execution Provider Operator Conformance Suite + +Status: Detailed design supporting +[Test Ownership and Operator Conformance](test_ownership_and_conformance_workstream.md). + +## Purpose + +Define a reusable ONNX Runtime (ORT) operator conformance suite that can validate in-tree and external execution +providers (EPs) against common operator semantics. + +The suite should let an EP author test a released plugin without cloning or building the ORT repository. It should +also preserve the value of ORT's existing operator tests while separating portable test cases from ORT-private C++ +test infrastructure. + +This facility is general to all EPs. WebGPU is an initial consumer and a useful migration test case, not a special +case in the design. + +## Background + +PR [#25689](https://github.com/microsoft/onnxruntime/pull/25689) created `onnxruntime_provider_test`, moved provider +and operator tests into that executable, and allowed tests using `OpTester` or `ModelTester` to run with a dynamically +registered plugin EP. + +That established two useful foundations: + +- Provider tests can run separately from the main ORT unit-test executable. +- An EP can be selected at runtime instead of being statically known to each test. + +`onnxruntime_provider_test` remains an in-tree ORT test binary. Its tests link private framework, graph, optimizer, +provider, and test libraries, and many cases are expressed as compiled C++ code. External EP repositories cannot +consume it as a stable release interface. + +## Goals + +- Provide common operator correctness cases that can be run against any EP. +- Use public ORT APIs at the runner boundary. +- Prevent CPU fallback from producing false passes. +- Support dynamically loaded and statically linked plugin EPs with the same cases and result rules. +- Support native and browser/Wasm runners without changing case meaning. +- Publish a versioned conformance kit associated with an ORT release. +- Make unsupported cases, exclusions, and tolerance overrides explicit and reportable. +- Allow existing `OpTester` and `ModelTester` cases to migrate incrementally. + +## Non-goals + +- Replace all tests in `onnxruntime_provider_test`. +- Make `OpTester`, `ModelTester`, or other ORT-private test helpers a stable public C++ API. +- Validate provider-specific implementation details such as shaders, vendor libraries, caches, or device limits. +- Require every EP to implement every operator in the suite. +- Define ONNX operator semantics independently from the ONNX specification. + +## Proposed components + +The design separates case authorship, case distribution, and execution: + +1. **Case definitions** describe models, inputs, expected outputs, comparison rules, and requirements. +2. **Case generator** validates definitions and emits portable serialized models and datasets. +3. **Conformance kit** packages generated cases, schemas, documentation, and a native runner for an ORT release. +4. **Runner** registers an EP, executes selected cases, enforces fallback rules, and writes a structured report. +5. **Provider profile** declares the EP's expected support surface, options, exclusions, and narrow comparison + overrides. + +The portable contract is the case format and execution/result semantics. A particular C++ runner implementation is +not the contract. + +## Relationship to `onnxruntime_provider_test` + +`onnxruntime_provider_test` should remain the comprehensive in-tree provider regression executable. It can use +ORT-private helpers and cover implementation details that are inappropriate for an external contract. + +The conformance suite should be a second layer: + +| Layer | Purpose | Dependencies | +| --- | --- | --- | +| `onnxruntime_provider_test` | In-tree provider and operator regression testing | ORT-private libraries and helpers | +| `onnxruntime_ep_conformance_test` | Portable EP operator conformance | Public ORT APIs and released case data | + +During migration, an `OpTester` or `ModelTester` case may remain the authoring source while tooling exports an +equivalent portable case. Over time, reusable cases should have one canonical data-driven definition consumed by +both test layers where practical. + +## Case ownership and storage + +Canonical ORT and contrib-op case definitions should live in the ORT repository, for example: + +```text +onnxruntime/test/ep_conformance/ + cases/ + onnx/ + contrib/ + schemas/ + case.schema.json + provider-profile.schema.json + report.schema.json + tools/ + generate_cases.py + runner/ +``` + +Standard ONNX cases should reuse or derive from ONNX backend test data where practical. ORT owns cases for ORT contrib +operators and generic ORT EP behavior. + +Generated `.onnx` models and large tensor datasets do not normally need to be checked in. ORT CI and release jobs can +generate them into a conformance-kit archive. A model should be checked in only when its exact serialized form is +part of the test or generation is not reasonably deterministic. + +Provider repositories own their profiles, exclusions, and implementation-specific tests. For example: + +```text +webgpu-ep/tests/conformance/ + provider-profile.json + exclusions.json +``` + +## Case representation + +A simple single-operator case can use a compact declarative representation containing: + +- Stable case identifier. +- Operator domain, name, and opset version. +- Input and output names, types, shapes, and values. +- Operator attributes. +- Comparison policy. +- Required capabilities or environmental constraints. +- Execution requirements such as complete assignment to the target EP. + +For example: + +```json +{ + "id": "ai.onnx.Add.opset14.float32.broadcast", + "operator": { + "domain": "", + "type": "Add", + "opset": 14 + }, + "inputs": [ + {"name": "A", "type": "float32", "shape": [2, 3], "values": [1, 2, 3, 4, 5, 6]}, + {"name": "B", "type": "float32", "shape": [3], "values": [10, 20, 30]} + ], + "outputs": [ + {"name": "C", "type": "float32", "shape": [2, 3], "values": [11, 22, 33, 14, 25, 36]} + ], + "comparison": { + "rtol": 0.0001, + "atol": 0.00001, + "nan_equal": true + }, + "execution": { + "require_target_ep": true, + "allow_cpu_fallback": false + } +} +``` + +The release generator wraps such a case in a valid ONNX model. The runner passes serialized model bytes to a normal +ORT session, so the test exercises graph capability discovery and compilation as well as execution. + +Complex graphs, control flow, functions, external data, malformed models, and tests where exact protobuf structure +matters may use packaged `.onnx` models directly. Large or shared tensor values may use ONNX backend-test-style +`test_data_set_*` directories rather than inline values. + +## Execution semantics + +For an ordinary operator conformance case, the runner should: + +1. Load or register the requested plugin EP and select a device. +2. Apply the requested EP options. +3. Create a session with CPU fallback disabled. +4. Load the generated or packaged model. +5. Require the target EP to accept the nodes specified by the case. +6. Run every input dataset. +7. Compare every output according to the case's comparison policy. +8. Record diagnostics without changing the defined outcome. + +Disabling CPU fallback is essential. A correct result produced by the CPU EP does not demonstrate conformance of the +target EP. + +Cases that intentionally test partial graph assignment must state their assignment requirements explicitly. They +should be classified separately from single-operator correctness cases. + +## Result semantics + +Each selected case should produce exactly one result: + +- `PASS`: The target EP executed the required graph and all outputs matched. +- `UNSUPPORTED`: The EP did not claim a case outside its declared support profile. +- `FAIL`: Output mismatch, crash, timeout, unexpected rejection, or unexpected fallback. +- `EXCLUDED`: A provider exclusion matched the case and supplied a documented reason. +- `NOT_RUN`: The environment could not execute the case, such as when no compatible device was available. + +`UNSUPPORTED` is not automatically a passing result. If the provider profile says the case is supported, rejection +is a failure. `NOT_RUN` should make a required CI lane incomplete rather than successful. + +An exclusion should identify a stable case ID, a reason, and preferably a tracking issue. Broad wildcard skip lists +should be discouraged because they obscure coverage loss. + +## Comparison semantics + +The case owns the default comparison policy. The policy may specify: + +- Exact comparison. +- Absolute and relative tolerances. +- NaN and infinity handling. +- Type-specific rules. +- Ordering rules where the operator permits multiple valid orders. + +A provider may define a narrow override when implementation precision requires it. Every override should include a +reason and should be visible in the report. Provider-wide tolerance inflation should not be supported. + +## Provider profile + +A provider profile defines test expectations rather than replacing the EP's runtime capability implementation. It +may contain: + +- Plugin registration name and selected EP name. +- Device selection and EP options. +- Supported domains, opsets, data types, and optional features. +- Case tags to include or exclude from a particular environment. +- Documented exclusions. +- Narrow comparison overrides. + +The runner uses the profile to distinguish an expected lack of support from a regression in the provider's declared +support surface. + +## Distribution + +An ORT release should publish a versioned conformance kit, for example: + +```text +onnxruntime-ep-conformance-1.25.0/ + bin/ + onnxruntime_ep_conformance_test + cases/ + onnx/ + contrib/ + schemas/ + examples/ + VERSION +``` + +The native runner is platform-specific. The generated case archive and schemas should be platform-neutral. + +An external EP should test against at least: + +- The minimum ORT release it supports. +- The current ORT release used for packaging. +- An ORT `main` or nightly conformance kit as an early-warning lane. + +## Dynamic plugin usage + +A native dynamically loaded plugin could be tested as follows: + +```powershell +onnxruntime_ep_conformance_test ` + --ep-library .\onnxruntime_providers_webgpu.dll ` + --registration-name WebGPUExecutionProvider ` + --ep-name WebGPUExecutionProvider ` + --cases .\cases ` + --provider-profile .\provider-profile.json ` + --report .\results.json +``` + +The runner should use public plugin registration, device discovery, session creation, and execution APIs. + +## Static plugin usage + +A prebuilt executable cannot discover a statically linked plugin. ORT should therefore also publish a small runner +SDK or CMake target that allows an EP repository to supply static factory registration: + +```cmake +find_package(onnxruntime_ep_conformance CONFIG REQUIRED) + +add_executable(webgpu_ep_conformance static_ep_registration.cc) +target_link_libraries( + webgpu_ep_conformance + PRIVATE + onnxruntime::ep_conformance_runner + webgpu_ep_static +) +``` + +This executable should consume the same case archive and produce the same report as the dynamic runner. Static and +dynamic linkage must not create separate conformance definitions. + +## WebAssembly and browser usage + +`onnxruntime-web` cannot use the native dynamic-plugin executable. A JavaScript or browser runner should load the same +portable cases, invoke the statically registered WebGPU plugin through ORT Web, and produce results with the same +schema and outcome rules. + +Browser-specific scheduling, test sharding, and artifact loading are host concerns. They must not change the meaning +of `PASS`, `FAIL`, `UNSUPPORTED`, `EXCLUDED`, or `NOT_RUN`. + +## Report format + +Reports should be machine-readable and include enough provenance to reproduce a run: + +- Report schema version. +- Conformance-suite and ORT versions. +- EP name and version. +- Dynamic or static registration mode. +- Device and relevant environment information. +- Provider profile hash. +- Per-case result, duration, and diagnostics. +- Summary counts by result, domain, operator, opset, and data type. + +The report should make newly unsupported or newly excluded cases easy to detect in CI. + +## Migration approach + +### Phase 1: Define the contract + +- Define case, provider-profile, and report schemas. +- Implement a public-API-only native runner for dynamic plugin EPs. +- Enforce CPU-fallback prevention. +- Convert a small representative set of ONNX and contrib operators. +- Run those cases against CPU and at least one plugin EP. + +### Phase 2: Connect existing test infrastructure + +- Add an export path from suitable `OpTester` and `ModelTester` cases. +- Generate conformance cases in CI and verify deterministic output. +- Produce a coverage map from existing provider tests to conformance case IDs. +- Keep `onnxruntime_provider_test` authoritative until converted cases demonstrate parity. + +### Phase 3: Publish and consume release kits + +- Publish native runners and platform-neutral case archives with ORT releases. +- Add a nightly kit for ORT `main`. +- Add a static-runner SDK or CMake package. +- Integrate the released kit into an external plugin EP repository. + +### Phase 4: Expand coverage and hosts + +- Migrate broadly reusable operator cases. +- Add browser/Wasm execution of the same cases. +- Add conformance coverage reporting to ORT and EP CI. +- Retain implementation-specific tests in their owning repositories. + +## Initial success criteria + +- One case definition runs against CPU and a dynamically loaded plugin EP. +- The same case detects and fails unexpected CPU fallback. +- Dynamic and static forms of one plugin produce equivalent results. +- An external EP repository can run a released kit without an ORT source checkout. +- Results distinguish failures, unsupported cases, exclusions, and infrastructure failures. +- Existing provider coverage can be mapped to stable conformance case IDs without an all-at-once migration. + +## Open questions + +- Should canonical simple cases use JSON, protobuf, Python source, or another representation? +- Which ONNX backend cases can be consumed directly without duplication? +- What public mechanism best proves target-EP assignment when partial assignment is allowed? +- How should capability profiles express operator attributes and shape constraints without duplicating + `GetCapability()`? +- Should contrib-op cases ship in the default kit or a separate ORT-extension bundle? +- Which runner artifacts should be included in each ORT package and release channel? +- How should large datasets be deduplicated and downloaded? +- What compatibility promise applies to case, provider-profile, and report schema versions? +- How should randomized or generated inputs remain deterministic and reproducible? +- What is the minimum representative operator set needed before using the suite as an extraction prerequisite? diff --git a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md new file mode 100644 index 0000000000000..fcdcc0d91874a --- /dev/null +++ b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md @@ -0,0 +1,195 @@ +# Workstream `test-conformance`: Test Ownership and Operator Conformance + +Status: Working plan + +[WebGPU EP extraction overview](../webgpu_ep_extraction.md) + +## Objective + +Preserve WebGPU regression coverage during extraction, assign every test to its long-term owner, and establish a +portable operator conformance layer that an external provider repository can run through public ORT interfaces. + +The repository move must not wait for every reusable ORT test to be converted to a new format. It must wait until all +existing coverage is accounted for and continues to run against the isolated or external provider. + +The detailed general-purpose conformance design is in +[Execution Provider Operator Conformance Suite](ep_operator_conformance_design.md). + +## Desired end state + +- Every current WebGPU-related test has a stable owner and destination. +- No test disappears merely because its implementation moves repositories. +- Existing tests run against the plugin path before the legacy provider path is removed. +- WebGPU-specific implementation tests live with the provider. +- Portable operator correctness cases remain ORT-owned and are distributed in a versioned conformance kit. +- Generic plugin contract and ORT integration tests remain in ORT. +- Static and shared provider forms run equivalent operator cases and result rules. +- CPU fallback cannot produce a false pass. + +## Test classification + +Create an inventory covering C++, JavaScript/TypeScript, Python, package, browser, and CI-only tests. Assign each test +one primary class: + +| Class | Long-term owner | Examples | +| --- | --- | --- | +| Portable operator conformance | ORT | ONNX and contrib operator semantics, type and shape combinations | +| WebGPU-specific implementation | WebGPU repository | Shaders, Dawn behavior, device limits, caches, provider options, and EP-specific operator behavior | +| ORT/plugin integration | ORT | Registration, session integration, loading errors, generic lifecycle | +| Host integration | Owning host repository | ORT Web module assembly, generic Node loading, package-host behavior | +| Temporary legacy | Explicitly recorded | Existing private ORT test retained while equivalent portable or provider-owned coverage is being established | + +Each inventory entry should record: + +- Current test and CI lane. +- Behavior protected. +- Current provider path. +- Classification and future owner. +- Planned destination or conformance case ID. +- Replacement status. +- Required platforms or devices. +- Tracking issue for temporary legacy coverage. + +Classification is a prerequisite for deleting or moving tests, not for beginning other workstreams. + +A temporary legacy test is not a permanent ownership category. Its inventory entry must identify the intended final +class, replacement test or conformance case, tracking issue, and removal criteria. It remains blocking until the +replacement runs in all required lanes, after which the legacy test is removed. + +## Regression protection before extraction + +Before the source move: + +- Run existing WebGPU operator tests through the plugin adapter path. +- Disable or detect CPU fallback for cases intended to validate WebGPU. +- Preserve current platform and browser lanes or document an approved replacement. +- Establish baseline results for the extraction launch matrix in + [WebGPU EP Repository Extraction](../webgpu_ep_extraction.md). +- Make isolated-tree tests blocking before removing their in-tree originals. +- Verify package installation and execution for existing Python, NuGet, Node, and Web consumers as applicable. + +An existing private ORT test may remain temporarily authoritative if it executes the isolated provider. Conversion to +the portable conformance format can continue after extraction. + +## WebGPU-specific test migration + +Tests move with the provider when they validate implementation choices rather than portable operator semantics. +Likely categories include: + +- WGSL generation and shader compilation. +- Dawn backend selection and integration. +- Device features, limits, and adapter behavior. +- Buffer, pipeline, and query caching. +- Provider options and diagnostics. +- `GPUDevice` and `GPUBuffer` WebGPU-specific interop behavior. +- Device loss and WebGPU-specific lifetime behavior. +- Performance regressions and implementation-specific workarounds. +- Plugin package contents and installation. + +The `provider-isolation` workstream owns the physical relocation and external CI. This workstream defines the +classification and verifies that replacement coverage is equivalent. + +## Conformance MVP + +The initial conformance milestone should be deliberately bounded. It needs to prove the contract required for safe +extraction, not complete migration of ORT's operator test suite. + +The MVP should provide: + +- A native runner for dynamic plugin EPs whose execution boundary uses only released public ORT APIs. +- A static-runner target or SDK using the same case data. +- CPU fallback prevention or reliable assignment verification. +- Case, provider-profile, and report schemas. +- Structured `PASS`, `UNSUPPORTED`, `FAIL`, `EXCLUDED`, and `NOT_RUN` results. +- A representative set of ONNX and contrib cases spanning important data types, shapes, options, and failure modes. +- Execution against CPU as a reference and WebGPU in shared and static forms. +- A versioned artifact usable without an ORT source checkout. + +The representative set should be chosen from the current WebGPU support surface and include enough diversity to +exercise capability discovery, model loading, execution, output comparison, unsupported behavior, and fallback +detection. + +ORT may build and publish the native runner itself. Restricting its execution boundary to public APIs ensures an +external EP can use the released runner and case archive without an ORT source checkout or a dependency on ORT's +private C++ ABI. A small source or CMake SDK may additionally be published for statically linked runners. + +## Coverage continuity rules + +- A test may be deleted only after its replacement is blocking in the required CI lanes. +- A moved test must protect the same behavior and platforms unless a reduction is explicitly approved. +- An unsupported result is a failure when the provider profile declares support. +- `NOT_RUN` does not satisfy a required lane. +- Exclusions require stable case IDs, reasons, and preferably tracking issues. +- Provider-wide tolerance inflation is not an acceptable migration shortcut. +- Static and dynamic forms should share case definitions and outcome semantics. +- For cases requiring full target-EP assignment, use the existing `session.disable_cpu_ep_fallback` session option. + Cases intentionally allowing partial assignment need explicit assignment requirements and may require additional + reporting. +- Keep the inventory current while extraction is in progress. New WebGPU tests must be classified when added, and CI + should detect test files or registrations missing from the inventory where practical. + +## Parallel work packages + +1. **Inventory and classification:** enumerate tests and produce the ownership map. +2. **Plugin-path baseline:** run existing cases through the adapter and close fallback blind spots. +3. **WebGPU-specific relocation:** move provider-owned tests into the isolated staging root. +4. **Conformance schemas and runner:** implement the public execution and reporting contract. +5. **Representative case conversion:** convert a bounded extraction-gate set. +6. **External CI integration:** run existing and conformance coverage against clean provider artifacts. +7. **Coverage reporting:** detect newly unsupported, excluded, or unexecuted cases. + +Inventory, runner design, and provider-specific relocation can proceed concurrently. Static runner validation depends +on the `plugin-boundary` workstream's static registration facility. + +## Interfaces with other workstreams + +### Plugin boundary and Web/Wasm integration + +- Requires dynamic and static provider registration entry points. +- Supplies fallback detection and parity gates. +- Keeps generic plugin contract tests in ORT. + +### Provider isolation and repository migration + +- Supplies the test ownership map and required destinations. +- Requires standalone provider artifacts and CI environments. +- Moves provider profiles, exclusions, and implementation tests with the provider. + +### Node plugin migration + +- Supplies Node package installation and execution coverage for the launch matrix. +- Reuses portable cases where practical but keeps Node host-loading behavior in the Node workstream. + +## Extraction gates + +Extraction may proceed when: + +- Every existing WebGPU-related test is inventoried and classified. +- Existing blocking behavior continues to run against the isolated provider. +- WebGPU-specific tests have moved or have blocking equivalent coverage. +- ORT integration tests cover dynamic and static registration contracts. +- The conformance MVP runs a representative set against shared and static WebGPU. +- CPU fallback produces a failure for cases requiring WebGPU assignment. +- Existing supported consumers pass installation and execution tests. +- Remaining temporary legacy tests have owners and removal criteria. + +Complete conversion of all suitable `OpTester` and `ModelTester` cases is not an extraction gate. Loss of current +tested behavior is an extraction blocker. + +## Completion criteria + +- The ownership inventory contains no unclassified tests and remains current through the extraction cutover. +- The external provider CI protects WebGPU-specific behavior and current operator coverage. +- ORT publishes and consumes a usable conformance kit. +- Static and dynamic WebGPU reports are comparable and expose coverage regressions. +- Temporary legacy coverage is either removed or tracked with explicit exit criteria. +- Continued conformance expansion no longer requires coordinated provider source changes. + +## Open questions + +- What exact current test set defines the extraction regression baseline? +- Which operators and data types form the minimum representative conformance set? +- What additional reporting is needed for cases that intentionally permit partial graph assignment? +- Which browser tests can consume the same portable cases without changing semantics? +- How should large test datasets be versioned and distributed? +- Which test generators should remain in ORT, move, or be copied? diff --git a/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md b/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md new file mode 100644 index 0000000000000..65cdb3b990fd0 --- /dev/null +++ b/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md @@ -0,0 +1,148 @@ +# WebGPU EP Repository Extraction + +Status: Working plan + +## Goal + +Move the WebGPU Execution Provider (EP) into its own repository without making ongoing WebGPU EP development slower or +more fragile. + +The new repository should own the WebGPU EP implementation, dependencies, tests, optional plugin packages, release +process, and development workflow. ONNX Runtime (ORT) should consume versioned WebGPU EP artifacts or source rather +than remain the implementation repository. + +The WebGPU EP remains optional for native ORT consumers. Core ORT packages should support plugin loading but must not +bundle or depend on the WebGPU plugin package. `onnxruntime-web` is the exception because it must statically link the +provider into its WebAssembly module. + +## Guiding principles + +- Use the public plugin EP interface (`OrtApi`, `OrtEpApi`, `OrtEpFactory`, and `OrtEp`) as the only runtime boundary + between ORT and the WebGPU EP. +- Use the same provider implementation for dynamically loaded and statically linked builds. +- Keep the WebGPU EP independently buildable, testable, versioned, and releasable. +- A normal kernel change should be developed, tested, reviewed, and released in the WebGPU repository alone. +- Do not create an undocumented cross-repository C++ interface to private ORT implementation code. +- Treat copied ORT helpers as WebGPU-owned forks. Preserve provenance and license, but do not keep the copies + synchronized with ORT. +- Keep cross-repository integration reproducible through pinned versions, compatibility checks, and CI. +- Preserve existing tested behavior and supported consumers throughout the move. + +## Workstreams + +The effort is divided into four workstreams: + +| Identifier | Workstream | Primary outcome | +| --- | --- | --- | +| `plugin-boundary` | [Plugin boundary and Web/Wasm integration](plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md) | Static and dynamic builds use the same public plugin EP boundary, including the ORT Web browser bridge | +| `provider-isolation` | [Provider isolation and repository migration](provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md) | WebGPU-owned code, dependencies, tests, and existing plugin packaging move under `plugin-ep-webgpu/` and then to an independent repository | +| `test-conformance` | [Test ownership and operator conformance](test_ownership_and_conformance/test_ownership_and_conformance_workstream.md) | Existing coverage is preserved, every test has an owner, and portable conformance coverage protects the external provider | +| `node-migration` | [Node plugin migration](node_plugin_migration/node_plugin_migration_workstream.md) | Existing bundled Node WebGPU support is replaced by an explicitly consumable plugin without regressing current users | + +The detailed reusable conformance-suite design is in +[Execution Provider Operator Conformance Suite](test_ownership_and_conformance/ep_operator_conformance_design.md). + +## Workstream dependencies + +The workstreams can make progress concurrently, with these convergence points: + +- Provider isolation identifies private dependencies. A dependency becomes public API work only when an existing + public API cannot express a necessary, stable runtime interaction and the proposed addition meets the high bar for + a permanent plugin EP API. +- Test classification determines which tests move with the provider and which remain in ORT. +- Static plugin registration enables ORT Web parity and static conformance execution. +- The Node workstream consumes generic plugin loading from ORT and native WebGPU artifacts from the external provider. +- The final repository copy depends on `plugin-ep-webgpu/` being independently buildable. + +```text +Plugin boundary ───────────────> ORT Web static integration + │ + ├───────────────────────> Node plugin loading + │ +Provider isolation ────────────> External provider artifacts + │ │ + └──────────────────────────────> Node WebGPU package + +Test classification ──────────> Provider test relocation +Conformance MVP ──────────────> Static/shared parity gates +``` + +## Extraction launch matrix + +The extraction launch matrix records existing consumers that require an explicit disposition before built-in code is +removed. Platform and architecture details remain to be inventoried in the individual workstreams. + +| Consumer or package | Launch disposition | Extraction requirement | +| --- | --- | --- | +| `onnxruntime-web` | Consume a pinned external WebGPU source revision through static plugin registration | Required | +| Python WebGPU plugin package | Move the existing optional plugin package and release pipeline | Required | +| WebGPU NuGet plugin package | Move the existing optional plugin package and release pipeline | Required | +| Node WebGPU support | Provide a tested replacement for the WebGPU implementation currently bundled in `onnxruntime-node`; final package naming is open | Required before bundled support is removed | +| Deprecated native packages with built-in WebGPU | Retire according to an explicit compatibility plan; do not make core packages depend on or bundle the plugin | Required disposition | +| New native hosts without current WebGPU support | Add later unless explicitly included in the launch matrix | Not an extraction prerequisite | + +The matrix should be updated as the current package and platform inventory is completed. Any removal from the launch +scope requires an explicit compatibility decision rather than silently dropping support. + +## Integrated delivery stages + +### Stage 1: Establish baselines + +- Inventory private dependencies, build inputs, packages, consumers, platforms, and tests. +- Record every current test's owner and intended destination. +- Establish correctness, package installation, WebAssembly size, and startup baselines. +- Confirm the launch matrix and platform coverage. + +### Stage 2: Execute workstreams in parallel + +- Add generic static plugin registration and run WebGPU through the plugin path. +- Consolidate provider-owned sources and build inputs under `plugin-ep-webgpu/`. +- Preserve current tests while implementing the conformance MVP. +- Scaffold the external repository and CI against the standalone build contract. +- Design and implement the Node plugin-loading and package transition. + +### Stage 3: Prove extraction readiness + +The source move is ready when: + +- Static and dynamic forms execute the same provider implementation through the public plugin EP API. +- `plugin-ep-webgpu/` builds from a clean copy without private ORT implementation headers or libraries. +- Every existing test is classified and current tested behavior remains blocking. +- WebGPU-specific tests run from the isolated provider tree. +- A representative conformance set detects output failures, unsupported regressions, and CPU fallback. +- Existing Python and NuGet plugin packages can be produced from the isolated tree. +- Node has an approved package transition and a tested replacement path. +- `onnxruntime-web` passes browser integration, reduced-build, binary-size, and performance checks through static plugin + registration. + +### Stage 4: Move and switch consumers + +- Import the isolated provider code and selected history into the new repository. +- Publish versioned native plugin artifacts and immutable source artifacts for static consumers. +- Move Python and NuGet plugin releases to the WebGPU repository. +- Pin the WebGPU revision used by `onnxruntime-web`. +- Execute the approved Node package transition. +- Retire deprecated built-in packages and remove obsolete implementation and packaging code from ORT. + +## Success criteria + +- WebGPU has one runtime boundary with ORT: the public plugin EP API. +- Static and shared builds execute the same provider implementation and operator tests. +- `plugin-ep-webgpu/` can be copied and built outside the ORT source tree. +- The external repository owns WebGPU code, dependencies, tests, packages, and releases. +- ORT updates its pinned WebGPU revision through a routine dependency update. +- `onnxruntime-web` preserves supported functionality and accepted size and performance characteristics. +- Native ORT packages remain usable without installing WebGPU. +- Existing Node WebGPU users have a documented and tested migration path. +- Compatibility failures produce clear build-time, registration-time, or package-time diagnostics. + +## Cross-workstream decisions still open + +- What public API gaps remain after private dependencies are classified? +- What is the stable browser boundary for JavaScript `GPUDevice` and `GPUBuffer` objects? +- Which ORT-standard dependency mechanism should consume the external WebGPU source for WebAssembly builds? +- What platform and architecture matrix is required for the first independent release? +- What compatibility window should the WebGPU EP promise across ORT releases? +- Should the Node transition keep the `onnxruntime-node` name or introduce a core-only package such as + `onnxruntime-node-core`? +- How should coordinated ORT API and WebGPU provider changes be tested before either repository merges? From a795ec04b8793cd242edcff5e354bc8de8002144 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Mon, 17 Aug 2026 12:42:37 -0700 Subject: [PATCH 002/107] Clarify EP conformance runner architecture Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: df9d994f-d4ed-4e61-8104-57abb42ae205 --- ...test_ownership_and_conformance_workstream.md | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md index fcdcc0d91874a..7f9a8b1cad454 100644 --- a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md +++ b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md @@ -96,8 +96,9 @@ extraction, not complete migration of ORT's operator test suite. The MVP should provide: -- A native runner for dynamic plugin EPs whose execution boundary uses only released public ORT APIs. -- A static-runner target or SDK using the same case data. +- A shared conformance-runner core for case loading, execution, comparison, and reporting. +- A prebuilt native executable that uses the core and loads dynamic plugin EP libraries. +- A small SDK or CMake target that uses the same core and lets an external EP link its static factory registration. - CPU fallback prevention or reliable assignment verification. - Case, provider-profile, and report schemas. - Structured `PASS`, `UNSUPPORTED`, `FAIL`, `EXCLUDED`, and `NOT_RUN` results. @@ -109,9 +110,15 @@ The representative set should be chosen from the current WebGPU support surface exercise capability discovery, model loading, execution, output comparison, unsupported behavior, and fallback detection. -ORT may build and publish the native runner itself. Restricting its execution boundary to public APIs ensures an -external EP can use the released runner and case archive without an ORT source checkout or a dependency on ORT's -private C++ ABI. A small source or CMake SDK may additionally be published for statically linked runners. +The dynamic and static runners should be thin registration frontends over the same runner core so that linkage mode +does not change case interpretation or result semantics. + +The shared runner core must interact with ORT through released public APIs. The dynamic executable could technically +link ORT-private test libraries because ORT builds and distributes it as a self-contained binary. The same core, +however, must also be consumable by an external EP repository to build a statically linked runner. Requiring private +ORT headers or libraries there would couple the external repository to ORT's source layout and private C++ ABI. +Keeping the shared core on the public API boundary allows ORT to distribute one implementation for both frontends and +lets an external EP build the static form against a released ORT SDK without an ORT source checkout. ## Coverage continuity rules From e26dc644f20c2d8c848f565a96cec572ce076d38 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Mon, 17 Aug 2026 14:48:53 -0700 Subject: [PATCH 003/107] Refine WebGPU extraction lifecycle plans Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: df9d994f-d4ed-4e61-8104-57abb42ae205 --- ...boundary_and_web_integration_workstream.md | 45 ++++++++++++++++--- ...ion_and_repository_migration_workstream.md | 20 +++++++++ .../webgpu_ep_extraction.md | 1 + 3 files changed, 59 insertions(+), 7 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md index c437a559aba74..7936bb0f30434 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md @@ -48,6 +48,34 @@ The design must address: The facility must be generic and validated with at least one non-WebGPU test plugin where practical. +### Process-global ownership and teardown + +Static registration removes the dynamic-library unload boundary. The current WebGPU plugin cleanup path cannot be +reused unchanged: releasing a factory clears global WebGPU contexts and kernel registries, destroys a global logger +wrapper, and shuts down protobuf. In a statically linked process, those subsystems may still be used by ORT, another +factory, or another static plugin. + +Before static WebGPU parity is considered complete, classify every process-global subsystem as: + +- Factory-owned and safe to release with that factory. +- Provider-registration-owned and released after the last factory and session using that registration. +- Host-owned and never finalized by the provider. + +The static registration and provider lifetime design must cover: + +- Multiple factories and devices from one registration. +- Multiple static plugin registrations in one process. +- Duplicate registration and partial initialization failures. +- ORT environment and session teardown ordering. +- Browser worker and process-exit behavior. +- Reference counting where provider-global state is shared. +- Logging lifetime without invalidating the host logger. +- Protobuf lifetime without calling `ShutdownProtobufLibrary()` on host-owned state. +- WebGPU context and kernel-registry caches without invalidating live sessions or factories. + +`ReleaseEpFactory` must release only state whose ownership and last-user condition are established. A prototype that +executes correctly but retains unsafe dynamic cleanup behavior does not satisfy static parity. + ### WebGPU plugin-path parity Compile the existing WebGPU adapter implementation as both a shared plugin and a static plugin library. Exercise the @@ -62,7 +90,7 @@ Parity work includes: - Provider option behavior. - GPU tensor and buffer interoperability. - Error and diagnostic behavior. -- Process and session lifetime behavior. +- Process, environment, factory, and session lifetime behavior. The direct in-tree WebGPU provider path remains only as a temporary comparison baseline. @@ -105,12 +133,13 @@ The bridge should not expose unrelated ORT private implementation details. ## Parallel work packages 1. **Static registration core:** implement and contract-test generic static factory registration. -2. **Emscripten prototype:** compile the adapter path statically and run a small model. -3. **Gap inventory triage:** convert private-dependency findings into public API or provider-owned actions. -4. **Browser bridge:** specify and prototype object and lifetime exchange. -5. **Parity and retirement:** run the existing suite through the plugin path and remove the legacy path. +2. **Global lifetime contract:** inventory process-global state and implement safe ownership and teardown rules. +3. **Emscripten prototype:** compile the adapter path statically and run a small model. +4. **Gap inventory triage:** convert private-dependency findings into public API or provider-owned actions. +5. **Browser bridge:** specify and prototype object and lifetime exchange. +6. **Parity and retirement:** run the existing suite through the plugin path and remove the legacy path. -The first four packages can proceed largely in parallel. Legacy-path removal waits for their convergence. +The first five packages can proceed largely in parallel. Legacy-path removal waits for their convergence. ## Interfaces with other workstreams @@ -129,6 +158,9 @@ The first four packages can proceed largely in parallel. Legacy-path removal wai ## Completion criteria - Static and dynamic WebGPU builds use the same provider implementation and public API boundary. +- Factory release and environment teardown cannot shut down process-global state still owned or used by ORT, another + factory, or another plugin. +- Static WebGPU does not shut down host-owned protobuf or logging state. - A WebGPU model executes through static registration in `onnxruntime-web`. - Existing plugin-path tests are blocking and detect fallback. - All required private-runtime interactions have a documented public API or provider-owned replacement. @@ -139,5 +171,4 @@ The first four packages can proceed largely in parallel. Legacy-path removal wai ## Open questions - Should static factories be registered before environment creation or through environment construction options? -- How should process-global plugin state be finalized in static builds? - What is the stable representation of JavaScript-owned WebGPU objects at the C API boundary? diff --git a/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md b/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md index 5b0d92cc7b44d..0bb483ea5c515 100644 --- a/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md +++ b/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md @@ -72,6 +72,26 @@ During isolation: - Make generated files and downloaded dependencies explicit build outputs or inputs. - Support an ORT build override pointing at an adjacent WebGPU checkout. +### History migration + +Create a history-migration manifest that lists every current and historical path whose changes should be retained. +The implementation has moved across multiple ORT directories, and path filtering does not automatically follow every +rename. Filtering only the final `plugin-ep-webgpu/` subtree would therefore omit earlier provider history. + +Use `git filter-repo` or equivalent tooling to: + +- Select the complete historical path set. +- Remap those paths into the new repository layout. +- Retain relevant authors, dates, commit messages, branches, tags, and merge relationships where practical. +- Exclude unrelated ORT source and history. + +The filtered import rewrites commit IDs. Pull requests, reviews, issues, and other GitHub metadata are not Git objects +and do not transfer with repository history. Record the source ORT repository, extraction commit, filtering command or +script, and path manifest in the new repository so commits can be traced back to their original context. + +Perform and review a trial history import before the final source move. Verify representative files with `git log` +and blame, and confirm that the resulting repository does not contain unrelated or sensitive content. + ## Private dependency removal Create a machine-reviewable inventory of: diff --git a/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md b/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md index 65cdb3b990fd0..08afa57e2a3a2 100644 --- a/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md +++ b/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md @@ -106,6 +106,7 @@ scope requires an explicit compatibility decision rather than silently dropping The source move is ready when: - Static and dynamic forms execute the same provider implementation through the public plugin EP API. +- Process-global initialization and teardown in the static plugin setup work correctly. - `plugin-ep-webgpu/` builds from a clean copy without private ORT implementation headers or libraries. - Every existing test is classified and current tested behavior remains blocking. - WebGPU-specific tests run from the isolated provider tree. From 6b7d91953db08730eb131e12f4b60e36ce2967a9 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 20 Aug 2026 14:06:29 -0700 Subject: [PATCH 004/107] Clarify test ownership and align workstream doc structure Scope browser test relocation to provider behavior and keep ORT Web module assembly, packaging, and static registration tests in ORT. Record that browser-hosted provider tests are an exception to the no-source-checkout build contract, and that some web lanes are currently non-blocking or build-only. Drop the duplicated test classification in favor of the test-conformance doc, and exclude the deprecated JSEP TypeScript path from scope. Add a minimum-version validation work package so the declared MIN_ONNXRUNTIME_VERSION floor is verified rather than claimed, and correct the packaging migration scope to the pipeline definitions still outside the staging root. Align the four workstream docs: consistent section ordering, rename 'Parallel work packages' to 'Work packages' with explicit sequencing, flatten the plugin-boundary Scope wrapper, and separate provider-isolation versioning content from its open questions. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .../node_plugin_migration_workstream.md | 2 +- ...boundary_and_web_integration_workstream.md | 14 +- ...ion_and_repository_migration_workstream.md | 168 ++++++++++-------- ...st_ownership_and_conformance_workstream.md | 5 +- 4 files changed, 101 insertions(+), 88 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/node_plugin_migration/node_plugin_migration_workstream.md b/docs/design/webgpu_ep_extraction/node_plugin_migration/node_plugin_migration_workstream.md index 737f2f009323a..c861dd44e1fea 100644 --- a/docs/design/webgpu_ep_extraction/node_plugin_migration/node_plugin_migration_workstream.md +++ b/docs/design/webgpu_ep_extraction/node_plugin_migration/node_plugin_migration_workstream.md @@ -102,7 +102,7 @@ Required coverage includes: - Node worker initialization and cleanup. - Upgrade tests for the selected package transition. -## Parallel work packages +## Work packages 1. **API design:** define generic plugin registration for the singleton Node ORT environment. 2. **Binding implementation:** load and register arbitrary plugin EP libraries. diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md index 7936bb0f30434..fad16ccf24b6f 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md @@ -29,9 +29,7 @@ does not own WebGPU kernels, Dawn, shader tooling, or the provider's standalone - Native hosts can load WebGPU as an optional plugin without core ORT packages depending on it. - The legacy direct/static WebGPU provider path is removed after parity is demonstrated. -## Scope - -### Generic static plugin registration +## Generic static plugin registration Add an ORT facility that accepts statically linked plugin factory entry points. It should reuse the existing dynamic plugin path after library loading and symbol lookup. @@ -48,7 +46,7 @@ The design must address: The facility must be generic and validated with at least one non-WebGPU test plugin where practical. -### Process-global ownership and teardown +## Process-global ownership and teardown Static registration removes the dynamic-library unload boundary. The current WebGPU plugin cleanup path cannot be reused unchanged: releasing a factory clears global WebGPU contexts and kernel registries, destroys a global logger @@ -76,7 +74,7 @@ The static registration and provider lifetime design must cover: `ReleaseEpFactory` must release only state whose ownership and last-user condition are established. A prototype that executes correctly but retains unsafe dynamic cleanup behavior does not satisfy static parity. -### WebGPU plugin-path parity +## WebGPU plugin-path parity Compile the existing WebGPU adapter implementation as both a shared plugin and a static plugin library. Exercise the same factory, device discovery, provider options, allocator, data transfer, graph assignment, and execution code in @@ -94,7 +92,7 @@ Parity work includes: The direct in-tree WebGPU provider path remains only as a temporary comparison baseline. -### Plugin API gap closure +## Plugin API gap closure Provider-isolation work will identify uses of private ORT interfaces. Each finding should be resolved by: @@ -114,7 +112,7 @@ Likely investigation areas include: - Logging, threading, allocators, and data transfer. - Environment and process-global initialization. -### Browser/Wasm bridge +## Browser/Wasm bridge Define the smallest stable interface needed to pass JavaScript-owned WebGPU objects between ORT Web and the provider. @@ -130,7 +128,7 @@ The provider repository should own WebGPU-specific behavior. The boundary must d The bridge should not expose unrelated ORT private implementation details. -## Parallel work packages +## Work packages 1. **Static registration core:** implement and contract-test generic static factory registration. 2. **Global lifetime contract:** inventory process-global state and implement safe ownership and teardown rules. diff --git a/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md b/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md index 0bb483ea5c515..9eeff1a08bd4e 100644 --- a/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md +++ b/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md @@ -21,7 +21,7 @@ The WebGPU repository owns: - WebGPU-owned support code copied or replaced from ORT. - Dawn selection, patches, and build configuration. - WGSL templates, generators, and generated-source policy. -- WebGPU-specific unit, integration, browser, package, and regression tests. +- WebGPU-specific unit, integration, package, and regression tests, and browser tests of provider behavior. - Shared and static provider build targets. - Existing Python and NuGet plugin packaging and release pipelines. - Version metadata, compatibility policy, CI, and release artifacts. @@ -29,6 +29,9 @@ The WebGPU repository owns: ORT should consume versioned artifacts or source and should not remain an implementation or packaging repository for the provider. +The deprecated JSEP TypeScript compute path in `js/web/lib/wasm/jsep/` is out of scope. It is being removed rather +than moved. + ## Isolation strategy Expand the existing `plugin-ep-webgpu/` directory into the in-tree staging root. It already owns Python and NuGet @@ -64,34 +67,13 @@ elsewhere in the ORT tree. During isolation: -- Preserve selected history in the new repository through a filtered-history import, such as `git filter-repo` or an - equivalent subtree export. Moving files into `plugin-ep-webgpu/` first makes the ownership boundary and subsequent - history extraction clearer. The import should exclude unrelated ORT history where practical. +- Move files into `plugin-ep-webgpu/` first, which clarifies the ownership boundary and simplifies the later history + extraction described in History migration. - Avoid changing behavior merely to change ownership. - Keep ORT integration shims outside the provider root. - Make generated files and downloaded dependencies explicit build outputs or inputs. - Support an ORT build override pointing at an adjacent WebGPU checkout. -### History migration - -Create a history-migration manifest that lists every current and historical path whose changes should be retained. -The implementation has moved across multiple ORT directories, and path filtering does not automatically follow every -rename. Filtering only the final `plugin-ep-webgpu/` subtree would therefore omit earlier provider history. - -Use `git filter-repo` or equivalent tooling to: - -- Select the complete historical path set. -- Remap those paths into the new repository layout. -- Retain relevant authors, dates, commit messages, branches, tags, and merge relationships where practical. -- Exclude unrelated ORT source and history. - -The filtered import rewrites commit IDs. Pull requests, reviews, issues, and other GitHub metadata are not Git objects -and do not transfer with repository history. Record the source ORT repository, extraction commit, filtering command or -script, and path manifest in the new repository so commits can be traced back to their original context. - -Perform and review a trial history import before the final source move. Verify representative files with `git log` -and blame, and confirm that the resulting repository does not contain unrelated or sensitive content. - ## Private dependency removal Create a machine-reviewable inventory of: @@ -126,11 +108,10 @@ with the ORT implementation. The isolated subtree should build against a declared ORT plugin SDK or installed ORT package without an ORT source checkout. -It should produce: +The subtree build should produce: - A native shared plugin library. - A static plugin-API library for Emscripten and other static hosts. -- Provider metadata sufficient to diagnose ORT compatibility. - Test executables or packages that consume public ORT interfaces. - Development artifacts from the same commit as release artifacts. @@ -143,6 +124,41 @@ The build should support: - Reproducible source archives. - Adjacent-checkout development from ORT. +Browser-hosted provider tests are an exception to the no-source-checkout rule. Static linkage requires building ORT +Web with the provider, so the WebGPU repository builds ORT Web from a pinned ORT source revision using the +adjacent-checkout override. ORT separately validates its pinned WebGPU revision as part of ORT Web release gating. + +## Packaging migration + +Python and NuGet plugin packaging sources already live under `plugin-ep-webgpu/`. What remains outside the staging +root is the pipeline definitions that drive them, currently under `tools/ci_build/`. This workstream moves those +rather than redesigning the packaging. + +The packaging model is: + +- Core ORT packages do not depend on or bundle WebGPU. +- Users install the WebGPU plugin package separately. +- Plugin packages declare compatible ORT versions and fail clearly on incompatibility. +- Deprecated packages with built-in WebGPU are retired instead of being converted into plugin-dependent packages. +- `onnxruntime-web` consumes an immutable source archive or commit for static linkage. + +The Node workstream consumes the native shared plugin artifacts produced here. Node package naming, plugin +registration, and compatibility behavior are owned end-to-end by +[Node plugin migration](../node_plugin_migration/node_plugin_migration_workstream.md). + +## Test relocation + +The `test-conformance` workstream owns test classification and gating policy. This workstream physically relocates +the tests it classifies as WebGPU-owned and supplies the targets and environments they need: + +- Moving test sources and data into the staging root and then the external repository. +- Building provider test targets against a plugin SDK rather than the ORT build graph. +- Providing CI environments, devices, and browser hosts for the relocated lanes. +- Retaining in-tree originals until their relocated equivalents run. + +The classification table, coverage continuity rules, and extraction gates are in +[Test ownership and operator conformance](../test_ownership_and_conformance/test_ownership_and_conformance_workstream.md). + ## Repository foundation The external repository skeleton can be created before isolation is complete to validate: @@ -155,55 +171,58 @@ The external repository skeleton can be created before isolation is complete to - Issue ownership and contribution policy. - Release automation. -The final code import should wait until a clean copy of the staging root builds and tests without undeclared ORT -source-tree dependencies. +## History migration -## Packaging migration +Create a history-migration manifest that lists every current and historical path whose changes should be retained. +The implementation has moved across multiple ORT directories, and path filtering does not automatically follow every +rename. Filtering only the final `plugin-ep-webgpu/` subtree would therefore omit earlier provider history. -Python and NuGet plugin packaging already exist. This workstream migrates their sources and release pipelines rather -than redesigning them. +Use `git filter-repo` or equivalent tooling to: -The packaging model is: +- Select the complete historical path set. +- Remap those paths into the new repository layout. +- Retain relevant authors, dates, commit messages, branches, tags, and merge relationships where practical. +- Exclude unrelated ORT source and history. -- Core ORT packages do not depend on or bundle WebGPU. -- Users install the WebGPU plugin package separately. -- Plugin packages declare compatible ORT versions and fail clearly on incompatibility. -- Deprecated packages with built-in WebGPU are retired instead of being converted into plugin-dependent packages. -- `onnxruntime-web` consumes an immutable source archive or commit for static linkage. +The filtered import rewrites commit IDs. Pull requests, reviews, issues, and other GitHub metadata are not Git objects +and do not transfer with repository history. Record the source ORT repository, extraction commit, filtering command or +script, and path manifest in the new repository so commits can be traced back to their original context. -The Node workstream consumes the native shared plugin artifacts produced here. Node package naming, plugin -registration, and compatibility behavior are owned end-to-end by -[Node plugin migration](../node_plugin_migration/node_plugin_migration_workstream.md). +Perform and review a trial history import before the final source move. Verify representative files with `git log` +and blame, and confirm that the resulting repository does not contain unrelated or sensitive content. -## Test relocation +## Versioning and provenance -The `test-conformance` workstream classifies existing tests. This workstream physically moves tests classified as -WebGPU-owned, including: +The WebGPU EP version is independent from the ORT version. Compatibility metadata declares the minimum and tested ORT +versions instead of coupling release numbers. -- Kernel and shader generation tests. -- Dawn integration and backend tests. -- Device-limit and feature tests. -- Caching and provider-option tests. -- Browser interop tests for WebGPU-specific behavior. -- Package installation and artifact tests. -- Provider implementation regressions. +Package signing, provenance, and release controls should meet the same requirements as comparable ORT core packages. -Portable operator cases and generic plugin contract tests remain owned by ORT. Temporary legacy tests may remain in -ORT until equivalent external coverage is blocking. +ORT should consume WebGPU source using the standard mechanism used for comparable third-party source dependencies. +The dependency inventory should compare existing ORT mechanisms before selecting the exact implementation. -## Parallel work packages +## Work packages 1. **Dependency inventory:** enumerate includes, libraries, generated inputs, and build assumptions. 2. **Staging-root design:** define layout, targets, SDK inputs, and integration shims. 3. **Support-code isolation:** copy or replace implementation helpers. 4. **Dependency ownership:** move Dawn, patches, and WGSL generation. 5. **Standalone build:** produce static and shared artifacts outside the ORT build graph. -6. **Repository and CI scaffold:** validate clean-checkout development and release jobs. -7. **Packaging migration:** move the existing Python and NuGet pipelines. -8. **Source transfer:** copy the proven staging root and switch ORT to pinned consumption. - -Inventory, repository scaffolding, packaging analysis, and test classification can begin immediately. Source transfer -depends on the standalone build. +6. **Minimum-version validation:** run the provider against its declared `MIN_ONNXRUNTIME_VERSION` runtime so the + floor is verified rather than claimed. +7. **Repository and CI scaffold:** validate clean-checkout development and release jobs. +8. **Packaging migration:** move the plugin packaging pipeline definitions into the staging root. +9. **Source transfer:** copy the proven staging root, import filtered history, and switch ORT to pinned consumption. + +Sequencing: + +- Dependency inventory, staging-root design, dependency ownership, and repository and CI scaffold can start + immediately and proceed in parallel. +- Support-code isolation depends on the dependency inventory. +- The standalone build depends on staging-root design, support-code isolation, and dependency ownership. +- Minimum-version validation and packaging migration depend on the standalone build for their final form, but both + can be prototyped against current in-tree artifacts. +- Source transfer is the final package and depends on all of the others. ## Interfaces with other workstreams @@ -215,9 +234,8 @@ depends on the standalone build. ### Test ownership and operator conformance -- The `test-conformance` workstream decides test ownership and minimum regression gates. -- This workstream supplies external test targets and CI environments. -- The provider profile, exclusions, and WebGPU-specific tests move with the provider. +- The `test-conformance` workstream owns test classification and gating policy. +- This workstream supplies external test targets, CI environments, and browser hosts. ### Node plugin migration @@ -226,26 +244,22 @@ depends on the standalone build. ## Completion criteria -- All WebGPU-owned build inputs live under the isolated staging root. -- Shared and static libraries build from a clean copy without private ORT libraries or headers. -- Dawn and WGSL generation are provider-owned and reproducible. -- WebGPU-specific tests run from the isolated tree and remain blocking. -- Python and NuGet plugin packages are produced and tested from the isolated tree. -- The external repository CI builds, tests, and packages a clean checkout. -- ORT can consume a pinned external source artifact and can override it with an adjacent checkout. -- Native ORT packages remain WebGPU-independent. +### Isolation milestone -## Versioning, provenance, and remaining questions +The staging root is ready for transfer when a clean copy of it builds, tests, and packages everything listed in +Desired end state, with no provider build input resolving outside the root and no private ORT headers or libraries in +the link interface. -The WebGPU EP version is independent from the ORT version. Compatibility metadata declares the minimum and tested ORT -versions instead of coupling release numbers. +### End state -Package signing, provenance, and release controls should meet the same requirements as comparable ORT core packages. +The provider lives in its own repository: -ORT should consume WebGPU source using the standard mechanism used for comparable third-party source dependencies. -The dependency inventory should compare existing ORT mechanisms before selecting the exact implementation. +- The external repository CI builds, tests, and packages a clean checkout. +- ORT can consume a pinned external source artifact and can override it with an adjacent checkout. +- ORT retains provider integration shims only, not provider implementation, build, or packaging inputs. +- Native ORT packages remain WebGPU-independent. -Remaining questions: +## Open questions - Which copied ORT helpers need independent namespaces or API cleanup before transfer? - How should reduced-operator configuration be represented as an external provider input? diff --git a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md index 7f9a8b1cad454..ef30ee49e2771 100644 --- a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md +++ b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md @@ -62,7 +62,8 @@ Before the source move: - Run existing WebGPU operator tests through the plugin adapter path. - Disable or detect CPU fallback for cases intended to validate WebGPU. -- Preserve current platform and browser lanes or document an approved replacement. +- Preserve current platform and browser lanes or document an approved replacement. Some WebGPU web lanes are + currently non-blocking or build-only, so preserving them does not by itself establish a gate. - Establish baseline results for the extraction launch matrix in [WebGPU EP Repository Extraction](../webgpu_ep_extraction.md). - Make isolated-tree tests blocking before removing their in-tree originals. @@ -135,7 +136,7 @@ lets an external EP build the static form against a released ORT SDK without an - Keep the inventory current while extraction is in progress. New WebGPU tests must be classified when added, and CI should detect test files or registrations missing from the inventory where practical. -## Parallel work packages +## Work packages 1. **Inventory and classification:** enumerate tests and produce the ownership map. 2. **Plugin-path baseline:** run existing cases through the adapter and close fallback blind spots. From 0b96cbf0de3d115c31e11833dd94c8b892b4b9d2 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 20 Aug 2026 18:33:39 -0700 Subject: [PATCH 005/107] Address plan review: critical path, work package structure, terminology State the critical path explicitly. Static plugin registration is a prerequisite for removing the adapter, not merely an enabler, so it gates code isolation. Reflect this in the overview, the dependency diagram, Stage 2, and the provider-isolation sequencing list. Replace the misaligned ASCII diagram with Mermaid. Restructure provider-isolation work packages from nine to seven. Add a mechanical staging-root move that is off the critical path, since relocating files does not change which base classes the provider uses. Merge code isolation with the standalone build, because the provider is not independently buildable until the kernel-authoring foundation is replaced. Dissolve the dependency-ownership and packaging-migration packages into the packages that do the work: WebGPU owns Dawn and WGSL outright, and the packaging and CI pipelines must be rewired atomically with the build switch to avoid a broken window. Correct terminology and inaccuracies: distinguish the plugin EP API from ORT package build inputs, scope the JSEP exclusion to the WebGPU compute path so it does not imply WebNN removal, fix the WebGpuExecutionProvider name casing and show registration name and EP name as independent, replace a stale version in the conformance kit example, and cross-reference the result semantics rather than restating them. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- ...ion_and_repository_migration_workstream.md | 80 +++++++++++++------ .../ep_operator_conformance_design.md | 10 ++- ...st_ownership_and_conformance_workstream.md | 5 +- .../webgpu_ep_extraction.md | 40 ++++++---- 4 files changed, 87 insertions(+), 48 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md b/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md index 9eeff1a08bd4e..6c0141fdd9cce 100644 --- a/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md +++ b/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md @@ -23,14 +23,14 @@ The WebGPU repository owns: - WGSL templates, generators, and generated-source policy. - WebGPU-specific unit, integration, package, and regression tests, and browser tests of provider behavior. - Shared and static provider build targets. -- Existing Python and NuGet plugin packaging and release pipelines. +- Existing Python and NuGet plugin packaging, CI, and release pipelines. - Version metadata, compatibility policy, CI, and release artifacts. ORT should consume versioned artifacts or source and should not remain an implementation or packaging repository for the provider. -The deprecated JSEP TypeScript compute path in `js/web/lib/wasm/jsep/` is out of scope. It is being removed rather -than moved. +The deprecated JSEP TypeScript WebGPU compute path under `js/web/lib/wasm/jsep/` is out of scope. It is being removed +rather than moved. The WebNN code in that directory is unaffected by this work. ## Isolation strategy @@ -71,7 +71,9 @@ During isolation: extraction described in History migration. - Avoid changing behavior merely to change ownership. - Keep ORT integration shims outside the provider root. -- Make generated files and downloaded dependencies explicit build outputs or inputs. +- Make generated files and downloaded dependencies explicit build outputs or inputs. WGSL generation remains a + build-time step; the Python requirement it places on consuming builds is acceptable because ORT already requires + Python. - Support an ORT build override pointing at an adjacent WebGPU checkout. ## Private dependency removal @@ -89,7 +91,7 @@ Classify each dependency: | Resolution | Use when | | --- | --- | -| Existing public API | The plugin SDK already expresses the required runtime interaction | +| Existing public API | The plugin EP API already expresses the required runtime interaction | | New public plugin API | The operation is a stable, generally useful runtime boundary | | WebGPU-owned copy | The code is implementation support and can evolve independently | | WebGPU-specific replacement | Existing ORT code is unsuitable as a cross-repository dependency | @@ -105,8 +107,8 @@ with the ORT implementation. ## Standalone build contract -The isolated subtree should build against a declared ORT plugin SDK or installed ORT package without an ORT source -checkout. +The isolated subtree should build against the public ORT headers and an installed or pinned ORT package, without an +ORT source checkout. The subtree build should produce: @@ -117,7 +119,7 @@ The subtree build should produce: The build should support: -- Pinned and overridable ORT SDK locations. +- Pinned and overridable ORT package and header locations. - Pinned Dawn and other third-party dependencies. - Reduced-operator input from an ORT Web build. - Platform-specific symbol visibility and export rules. @@ -128,11 +130,21 @@ Browser-hosted provider tests are an exception to the no-source-checkout rule. S Web with the provider, so the WebGPU repository builds ORT Web from a pinned ORT source revision using the adjacent-checkout override. ORT separately validates its pinned WebGPU revision as part of ORT Web release gating. -## Packaging migration +## Packaging and pipeline migration Python and NuGet plugin packaging sources already live under `plugin-ep-webgpu/`. What remains outside the staging -root is the pipeline definitions that drive them, currently under `tools/ci_build/`. This workstream moves those -rather than redesigning the packaging. +root is the packaging and CI pipeline definitions that drive them, under `tools/ci_build/` and `.github/workflows/`. +This workstream relocates those and rewires them to invoke the standalone build, but does not redesign the packaging +scripts themselves. + +The pipelines invoke `tools/ci_build/build.py --use_webgpu shared_lib` and consume artifacts from ORT's build output +locations, so they break the moment the standalone build replaces the in-tree one. The rewiring therefore lands with +the standalone build rather than after it, and no temporary shim over `build.py` is maintained. Relocating the +pipeline files is mechanical and happens earlier, with the rest of the staging-root move. + +ORT workflows that merely trigger on `plugin-ep-webgpu/rel-*` branches stay in ORT and need only their trigger +configuration updated. Shared assets such as the WebGPU shader-key validation action are consumed by both moving and +staying lanes, so their ownership is decided individually. The packaging model is: @@ -152,7 +164,8 @@ The `test-conformance` workstream owns test classification and gating policy. Th the tests it classifies as WebGPU-owned and supplies the targets and environments they need: - Moving test sources and data into the staging root and then the external repository. -- Building provider test targets against a plugin SDK rather than the ORT build graph. +- Building provider test targets against the plugin EP API and an installed ORT package rather than the ORT build + graph. - Providing CI environments, devices, and browser hosts for the relocated lanes. - Retaining in-tree originals until their relocated equivalents run. @@ -204,24 +217,39 @@ The dependency inventory should compare existing ORT mechanisms before selecting ## Work packages 1. **Dependency inventory:** enumerate includes, libraries, generated inputs, and build assumptions. -2. **Staging-root design:** define layout, targets, SDK inputs, and integration shims. -3. **Support-code isolation:** copy or replace implementation helpers. -4. **Dependency ownership:** move Dawn, patches, and WGSL generation. -5. **Standalone build:** produce static and shared artifacts outside the ORT build graph. -6. **Minimum-version validation:** run the provider against its declared `MIN_ONNXRUNTIME_VERSION` runtime so the +2. **Staging-root design:** define layout, targets, ORT package inputs, and integration shims. +3. **Staging-root move:** relocate provider sources, Dawn patches, the WGSL templates and generator, and the plugin + packaging and CI pipeline definitions into the staging root without changing behavior. +4. **Code isolation and standalone build:** replace the kernel-authoring foundation with WebGPU-owned equivalents, + copy or replace the remaining implementation helpers, take over the Dawn dependency pin and fetch, produce static + and shared artifacts outside the ORT build graph, and rewire the packaging and CI pipelines onto that build. +5. **Minimum-version validation:** run the provider against its declared `MIN_ONNXRUNTIME_VERSION` runtime so the floor is verified rather than claimed. -7. **Repository and CI scaffold:** validate clean-checkout development and release jobs. -8. **Packaging migration:** move the plugin packaging pipeline definitions into the staging root. -9. **Source transfer:** copy the proven staging root, import filtered history, and switch ORT to pinned consumption. +6. **Repository and CI scaffold:** validate clean-checkout development and release jobs, including component + governance and compliance registration for the new repository. +7. **Source transfer:** copy the proven staging root, import filtered history, and switch ORT to pinned consumption. + +The staging-root move is deliberately separate and mechanical. It is one behavior-preserving relocation that +conflicts with in-flight WebGPU changes exactly once, and it lets later contributions land in the destination rather +than adding to the isolation work. + +Code isolation and the standalone build are one package because the provider is not independently buildable until the +kernel-authoring foundation is replaced, and that replacement is most of the work. The packaging and CI pipelines are +rewired in the same package because they drive the in-tree build directly and would otherwise break. Until the +separate build exists, isolation progress is visible in the WebGPU target's include and link lists in +`cmake/onnxruntime_providers_webgpu.cmake`; those lists are expected to shrink monotonically. Sequencing: -- Dependency inventory, staging-root design, dependency ownership, and repository and CI scaffold can start - immediately and proceed in parallel. -- Support-code isolation depends on the dependency inventory. -- The standalone build depends on staging-root design, support-code isolation, and dependency ownership. -- Minimum-version validation and packaging migration depend on the standalone build for their final form, but both - can be prototyped against current in-tree artifacts. +- Dependency inventory, staging-root design, and repository and CI scaffold can start immediately and proceed in + parallel. +- The staging-root move depends on staging-root design. It is not on the critical path, because relocating files does + not change which base classes the provider uses and the existing static build keeps working from the new location. +- Code isolation and standalone build depends on the dependency inventory and the staging-root move. It also depends + on generic static plugin registration from the `plugin-boundary` workstream, because the staging root must serve + the static build before the adapter can be removed. +- Minimum-version validation depends on code isolation and standalone build for its final form, but can be + prototyped against current in-tree artifacts. - Source transfer is the final package and depends on all of the others. ## Interfaces with other workstreams diff --git a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md index 6c2a6ffc1c852..b7c3f2f565e5b 100644 --- a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md +++ b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md @@ -228,7 +228,7 @@ support surface. An ORT release should publish a versioned conformance kit, for example: ```text -onnxruntime-ep-conformance-1.25.0/ +onnxruntime-ep-conformance-/ bin/ onnxruntime_ep_conformance_test cases/ @@ -254,14 +254,16 @@ A native dynamically loaded plugin could be tested as follows: ```powershell onnxruntime_ep_conformance_test ` --ep-library .\onnxruntime_providers_webgpu.dll ` - --registration-name WebGPUExecutionProvider ` - --ep-name WebGPUExecutionProvider ` + --registration-name webgpu_plugin ` + --ep-name WebGpuExecutionProvider ` --cases .\cases ` --provider-profile .\provider-profile.json ` --report .\results.json ``` -The runner should use public plugin registration, device discovery, session creation, and execution APIs. +The registration name is chosen by the caller and identifies the loaded library. The EP name is the one the factory +reports, and selects which provider from that library to use. The runner should use public plugin registration, +device discovery, session creation, and execution APIs. ## Static plugin usage diff --git a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md index ef30ee49e2771..822b00169e19a 100644 --- a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md +++ b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md @@ -102,7 +102,8 @@ The MVP should provide: - A small SDK or CMake target that uses the same core and lets an external EP link its static factory registration. - CPU fallback prevention or reliable assignment verification. - Case, provider-profile, and report schemas. -- Structured `PASS`, `UNSUPPORTED`, `FAIL`, `EXCLUDED`, and `NOT_RUN` results. +- Structured `PASS`, `UNSUPPORTED`, `FAIL`, `EXCLUDED`, and `NOT_RUN` results, with semantics defined in + [Execution Provider Operator Conformance Suite](ep_operator_conformance_design.md). - A representative set of ONNX and contrib cases spanning important data types, shapes, options, and failure modes. - Execution against CPU as a reference and WebGPU in shared and static forms. - A versioned artifact usable without an ORT source checkout. @@ -119,7 +120,7 @@ link ORT-private test libraries because ORT builds and distributes it as a self- however, must also be consumable by an external EP repository to build a statically linked runner. Requiring private ORT headers or libraries there would couple the external repository to ORT's source layout and private C++ ABI. Keeping the shared core on the public API boundary allows ORT to distribute one implementation for both frontends and -lets an external EP build the static form against a released ORT SDK without an ORT source checkout. +lets an external EP build the static form against a released ORT package without an ORT source checkout. ## Coverage continuity rules diff --git a/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md b/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md index 08afa57e2a3a2..b551809efa0a5 100644 --- a/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md +++ b/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md @@ -44,29 +44,33 @@ The detailed reusable conformance-suite design is in ## Workstream dependencies -The workstreams can make progress concurrently, with these convergence points: +Most work proceeds concurrently, but one sequence determines the end date: + +```mermaid +graph LR + A[plugin-boundary:
static plugin registration] --> B[provider-isolation:
code isolation and standalone build] + M[provider-isolation:
staging-root move] --> B + B --> C[provider-isolation:
source transfer] + D[test-conformance:
classification and conformance MVP] --> C + E[provider-isolation:
repository and CI scaffold] --> C + B -. native artifacts .-> G[node-migration:
Node WebGPU package] +``` + +Static plugin registration is a prerequisite for isolation, not merely an enabler. The `provider-isolation` workstream +moves provider sources into the staging root rather than copying them, so once isolation completes there is exactly +one WebGPU implementation and it is adapter-free. The non-plugin static build that `onnxruntime-web` and the +`onnxruntime-webgpu` package ship from today must already be served through static plugin registration before the +adapter can be removed. Relocating the sources is not itself blocked; removing the adapter is. + +The remaining convergence points are not on the critical path: - Provider isolation identifies private dependencies. A dependency becomes public API work only when an existing public API cannot express a necessary, stable runtime interaction and the proposed addition meets the high bar for a permanent plugin EP API. - Test classification determines which tests move with the provider and which remain in ORT. -- Static plugin registration enables ORT Web parity and static conformance execution. - The Node workstream consumes generic plugin loading from ORT and native WebGPU artifacts from the external provider. - The final repository copy depends on `plugin-ep-webgpu/` being independently buildable. -```text -Plugin boundary ───────────────> ORT Web static integration - │ - ├───────────────────────> Node plugin loading - │ -Provider isolation ────────────> External provider artifacts - │ │ - └──────────────────────────────> Node WebGPU package - -Test classification ──────────> Provider test relocation -Conformance MVP ──────────────> Static/shared parity gates -``` - ## Extraction launch matrix The extraction launch matrix records existing consumers that require an explicit disposition before built-in code is @@ -95,8 +99,12 @@ scope requires an explicit compatibility decision rather than silently dropping ### Stage 2: Execute workstreams in parallel +The critical path runs through this stage: static plugin registration, then code isolation and the standalone build. +The remaining items proceed alongside it. + - Add generic static plugin registration and run WebGPU through the plugin path. -- Consolidate provider-owned sources and build inputs under `plugin-ep-webgpu/`. +- Replace the kernel-authoring foundation and consolidate provider-owned sources and build inputs under + `plugin-ep-webgpu/`, so the staging root is adapter-free and independently buildable. - Preserve current tests while implementing the conformance MVP. - Scaffold the external repository and CI against the standalone build contract. - Design and implement the Node plugin-loading and package transition. From d20f696bef6e5e0b38ee8b438ec451d79a4cd167 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Fri, 21 Aug 2026 15:23:43 -0700 Subject: [PATCH 006/107] Address remaining plan review findings Adds a cross-repository version policy that resolves the two-way pin between ORT and the WebGPU repository, distinguishing the build-against ORT version from the runtime floor and keeping both blocking lanes on immutable released artifacts. Adds a Related work section stating independence from the JSEP migration, and a terminology table. Renames the extraction launch matrix to consumer dispositions and moves the non-consumer row to prose. Ties the size and inference latency gates to the Stage 1 baselines, removes the undefined startup metric, and adds a latency criterion to the plugin-boundary workstream. Records user-facing documentation ownership and post-cutover issue and pull request handling as part of the repository migration. Corrects a stale claim that onnxruntime-webgpu still ships from the non-plugin static build. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- ...boundary_and_web_integration_workstream.md | 16 +++-- ...ion_and_repository_migration_workstream.md | 72 +++++++++++++++++-- .../ep_operator_conformance_design.md | 15 ++-- ...st_ownership_and_conformance_workstream.md | 13 +++- .../webgpu_ep_extraction.md | 71 +++++++++++++----- 5 files changed, 156 insertions(+), 31 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md index fad16ccf24b6f..2ff7f53894775 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md @@ -76,9 +76,9 @@ executes correctly but retains unsafe dynamic cleanup behavior does not satisfy ## WebGPU plugin-path parity -Compile the existing WebGPU adapter implementation as both a shared plugin and a static plugin library. Exercise the -same factory, device discovery, provider options, allocator, data transfer, graph assignment, and execution code in -both forms. +The shared-library plugin form already builds and ships. The deliverable here is the static form for Emscripten and +other static hosts, plus lifetime correctness in both. Exercise the same factory, device discovery, provider options, +allocator, data transfer, graph assignment, and execution code in each. Parity work includes: @@ -111,6 +111,12 @@ Likely investigation areas include: - Reduced-operator configuration. - Logging, threading, allocators, and data transfer. - Environment and process-global initialization. +- Setting EP default configuration before a session exists, equivalent to the existing `SetCurrentGpuDeviceId`. + +The adapter's own `Missing parts` section in `onnxruntime/core/providers/webgpu/ep/README.md` is authoritative input +to this inventory rather than speculation. It records two gaps: WebGPU cleanup, which the process-global ownership +and teardown work covers, and EP default configuration, which is missing for both static and shared library builds +and sketches an `OrtApi` addition for it. ## Browser/Wasm bridge @@ -163,7 +169,9 @@ The first five packages can proceed largely in parallel. Legacy-path removal wai - Existing plugin-path tests are blocking and detect fallback. - All required private-runtime interactions have a documented public API or provider-owned replacement. - Browser object ownership and lifecycle are documented and tested. -- Reduced WebAssembly builds retain required plugin infrastructure within accepted size and startup budgets. +- Reduced WebAssembly builds retain required plugin infrastructure within accepted size budgets, measured against + the baselines established before the work begins. +- Inference latency through static plugin registration stays within an accepted tolerance of the same baselines. - The direct `IExecutionProvider` WebGPU path is removed. ## Open questions diff --git a/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md b/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md index 6c0141fdd9cce..9e290cadbeca9 100644 --- a/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md +++ b/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md @@ -25,6 +25,8 @@ The WebGPU repository owns: - Shared and static provider build targets. - Existing Python and NuGet plugin packaging, CI, and release pipelines. - Version metadata, compatibility policy, CI, and release artifacts. +- User-facing provider documentation, which onnxruntime.ai references rather than duplicates. +- WebGPU issues and pull requests. ORT should consume versioned artifacts or source and should not remain an implementation or packaging repository for the provider. @@ -41,14 +43,14 @@ ownership until the directory mirrors the intended external repository: ```text plugin-ep-webgpu/ cmake/ + patches/ + dawn/ include/ src/ ep/ kernels/ runtime/ support/ - third_party/ - dawn/ tools/ wgsl/ tests/ @@ -62,6 +64,9 @@ plugin-ep-webgpu/ MIN_ONNXRUNTIME_VERSION ``` +Dawn keeps its current form: a pinned, fetched dependency with patches applied at build time. The staging root holds +the pin and the patches, not a vendored copy of the source. + All provider-owned build inputs should be reachable from that root without consulting an implicit list of files elsewhere in the ORT tree. @@ -78,7 +83,20 @@ During isolation: ## Private dependency removal -Create a machine-reviewable inventory of: +The provider uses ORT-internal code deliberately. While the implementation and the runtime live in one repository, +reusing ORT's kernel-authoring types and operator helpers avoids duplicating code that is already maintained next to +it. That trade stops paying once the provider ships from its own repository, and unwinding it is a large part of what +the move is for. The coupling is not accidental, and removing it is not a correction. + +The provider's dependencies on ORT source fall into two groups with different dispositions: + +- **Framework surface** — the kernel-authoring types the provider is written against. Replacing these is the + kernel-authoring foundation work in the code isolation package, not a copy decision. +- **Operator helpers** — parameter parsing, shape math, and similar utilities shared with the CPU and CUDA EPs today + only because everything lives in one repository. Copying these is the intended outcome, and the provider owns the + correctness of its copies afterwards. + +The inventory is broader than source dependencies. Create a machine-reviewable inventory of: - Private ORT headers included by provider sources. - Private ORT libraries in provider link interfaces. @@ -128,7 +146,8 @@ The build should support: Browser-hosted provider tests are an exception to the no-source-checkout rule. Static linkage requires building ORT Web with the provider, so the WebGPU repository builds ORT Web from a pinned ORT source revision using the -adjacent-checkout override. ORT separately validates its pinned WebGPU revision as part of ORT Web release gating. +adjacent-checkout override. That revision is selected by the cross-repository version policy below. ORT separately +validates its pinned WebGPU revision as part of ORT Web release gating. ## Packaging and pipeline migration @@ -183,6 +202,11 @@ The external repository skeleton can be created before isolation is complete to - Artifact naming and retention. - Issue ownership and contribution policy. - Release automation. +- Component governance registration for Dawn and other dependencies, currently under `cgmanifests/webgpu/`. +- Security review, signing, and compliant release pipelines. + +The compliance items are long-lead in practice and are easy to defer until they block a release. Establishing them +with the skeleton keeps them off the critical path. ## History migration @@ -201,6 +225,10 @@ The filtered import rewrites commit IDs. Pull requests, reviews, issues, and oth and do not transfer with repository history. Record the source ORT repository, extraction commit, filtering command or script, and path manifest in the new repository so commits can be traced back to their original context. +After the cutover, WebGPU issues and pull requests belong in the new repository. Items open in `microsoft/onnxruntime` +at transfer time need a disposition: those describing provider behavior move or are refiled, while those describing +ORT-side integration stay where they are. + Perform and review a trial history import before the final source move. Verify representative files with `git log` and blame, and confirm that the resulting repository does not contain unrelated or sensitive content. @@ -214,6 +242,42 @@ Package signing, provenance, and release controls should meet the same requireme ORT should consume WebGPU source using the standard mechanism used for comparable third-party source dependencies. The dependency inventory should compare existing ORT mechanisms before selecting the exact implementation. +### Cross-repository version policy + +Each repository pins the other, so the pins must be arranged so that neither side can be blocked waiting for the +other. Only one lane floats, and it never blocks a merge: + +| Lane | Built or run against | Blocking | +| --- | --- | --- | +| WebGPU build | A released ORT version providing every EP API feature the provider references | Yes | +| WebGPU minimum-version validation | The declared `MIN_ONNXRUNTIME_VERSION` runtime | Yes | +| WebGPU integration | ORT main | No | +| ORT | Its pinned WebGPU revision | Yes | + +Both blocking WebGPU lanes target immutable released ORT artifacts, so neither can wait on an ORT revision that does +not yet exist. ORT advances its WebGPU pin on its own schedule and declines the update when it fails. + +Two ORT versions matter here, and they are not the same number: + +- The **build-against version** must declare every EP API the provider references, including calls reached only + through a runtime version gate, because a gated call still needs its declaration to compile. +- The **runtime floor**, `MIN_ONNXRUNTIME_VERSION`, is the oldest runtime the provider loads against. It can be + lower, because newer calls are gated at runtime through `CurrentOrtApiVersion()`. + +They coincide only while nothing is gated above the floor. A build against newer headers cannot detect a mis-gated +call, so the floor has to be exercised by running against it. That is the minimum-version validation package. + +The non-blocking integration lane against ORT main exists to catch ORT changes that break the provider while the fix +is still cheap. Neither blocking lane can do this: both target already-released ORT versions, so a regression +introduced on main stays invisible until it ships. + +A failure in the integration lane is an ORT compatibility regression and is fixed in ORT, because the boundary is the +public plugin EP API. The exception is a provider dependency on unspecified behavior, which is fixed in the WebGPU +repository. Without a stated owner the lane goes permanently red and stops being read. + +Adopting a newly added EP API therefore requires an ORT release carrying it before the provider can build against it. +This does not force the runtime floor upward, since the new call can be gated. + ## Work packages 1. **Dependency inventory:** enumerate includes, libraries, generated inputs, and build assumptions. diff --git a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md index b7c3f2f565e5b..304d1bf664ebb 100644 --- a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md +++ b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md @@ -60,8 +60,8 @@ The design separates case authorship, case distribution, and execution: 5. **Provider profile** declares the EP's expected support surface, options, exclusions, and narrow comparison overrides. -The portable contract is the case format and execution/result semantics. A particular C++ runner implementation is -not the contract. +The portable contract is the case format and execution/result semantics. A particular runner implementation is not +the contract, and the vehicle for the runner is still open — see Open questions. ## Relationship to `onnxruntime_provider_test` @@ -331,7 +331,7 @@ The report should make newly unsupported or newly excluded cases easy to detect - Publish native runners and platform-neutral case archives with ORT releases. - Add a nightly kit for ORT `main`. - Add a static-runner SDK or CMake package. -- Integrate the released kit into an external plugin EP repository. +- Integrate the conformance kit into an external plugin EP repository. ### Phase 4: Expand coverage and hosts @@ -345,12 +345,19 @@ The report should make newly unsupported or newly excluded cases easy to detect - One case definition runs against CPU and a dynamically loaded plugin EP. - The same case detects and fails unexpected CPU fallback. - Dynamic and static forms of one plugin produce equivalent results. -- An external EP repository can run a released kit without an ORT source checkout. +- An external EP repository can run a conformance kit without an ORT source checkout. - Results distinguish failures, unsupported cases, exclusions, and infrastructure failures. - Existing provider coverage can be mapped to stable conformance case IDs without an all-at-once migration. ## Open questions +- What should the runner be built on? Candidates are a new public-API-only C++ runner as sketched here, + `onnx_test_runner` extended with plugin EP registration, or a Python suite over the existing plugin registration + APIs. `onnx_test_runner` already ships, consumes ONNX backend test data, and supports disabling CPU fallback, but + links private ORT libraries. A Python suite has no build barrier for external consumers but covers native dynamic + loading only. +- Is a native static-linkage runner needed at all? The static-linkage consumer is `onnxruntime-web`, which requires a + browser runner regardless, so a native static host may be a hypothetical consumer. - Should canonical simple cases use JSON, protobuf, Python source, or another representation? - Which ONNX backend cases can be consumed directly without duplication? - What public mechanism best proves target-EP assignment when partial assignment is allowed? diff --git a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md index 822b00169e19a..8ecd25983ee3b 100644 --- a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md +++ b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md @@ -50,6 +50,12 @@ Each inventory entry should record: - Required platforms or devices. - Tracking issue for temporary legacy coverage. +While JSEP and the native WebGPU EP coexist, `onnxruntime-web` runs one browser test list against both. The same +suite executes against JSEP in the default and `./all` bundles and against the native WebGPU EP in the `./webgpu` and +`./jspi` bundles, selected when the bundle is built. Which implementation a browser test exercises is therefore a +property of the CI lane, not of the test, and does not change the test's class, owner, or destination. Record the +provider path per lane, and treat the JSEP lane as following JSEP removal rather than this extraction. + Classification is a prerequisite for deleting or moving tests, not for beginning other workstreams. A temporary legacy test is not a permanent ownership category. Its inventory entry must identify the intended final @@ -64,7 +70,7 @@ Before the source move: - Disable or detect CPU fallback for cases intended to validate WebGPU. - Preserve current platform and browser lanes or document an approved replacement. Some WebGPU web lanes are currently non-blocking or build-only, so preserving them does not by itself establish a gate. -- Establish baseline results for the extraction launch matrix in +- Establish baseline results for the consumers listed in [WebGPU EP Repository Extraction](../webgpu_ep_extraction.md). - Make isolated-tree tests blocking before removing their in-tree originals. - Verify package installation and execution for existing Python, NuGet, Node, and Web consumers as applicable. @@ -130,6 +136,9 @@ lets an external EP build the static form against a released ORT package without - `NOT_RUN` does not satisfy a required lane. - Exclusions require stable case IDs, reasons, and preferably tracking issues. - Provider-wide tolerance inflation is not an acceptable migration shortcut. +- Forking a shared ORT helper transfers the behavior it implements to the provider. Where that behavior was covered + only incidentally by tests of another consumer, the inventory entry must record whether existing coverage follows + the fork or new provider-side coverage is required. - Static and dynamic forms should share case definitions and outcome semantics. - For cases requiring full target-EP assignment, use the existing `session.disable_cpu_ep_fallback` session option. Cases intentionally allowing partial assignment need explicit assignment requirements and may require additional @@ -166,7 +175,7 @@ on the `plugin-boundary` workstream's static registration facility. ### Node plugin migration -- Supplies Node package installation and execution coverage for the launch matrix. +- Supplies Node package installation and execution coverage for the consumer dispositions. - Reuses portable cases where practical but keeps Node host-loading behavior in the Node workstream. ## Extraction gates diff --git a/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md b/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md index b551809efa0a5..a704ea097a5c3 100644 --- a/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md +++ b/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md @@ -2,6 +2,10 @@ Status: Working plan +This overview and the workstream documents it links define objectives, ownership, sequencing, and gates. +Implementation detail such as dependency inventories, staging-root layout, and test classification is produced by the +work packages themselves rather than specified here. + ## Goal Move the WebGPU Execution Provider (EP) into its own repository without making ongoing WebGPU EP development slower or @@ -24,10 +28,22 @@ provider into its WebAssembly module. - A normal kernel change should be developed, tested, reviewed, and released in the WebGPU repository alone. - Do not create an undocumented cross-repository C++ interface to private ORT implementation code. - Treat copied ORT helpers as WebGPU-owned forks. Preserve provenance and license, but do not keep the copies - synchronized with ORT. + synchronized with ORT. A plugin EP is responsible for the correctness of its own implementation, whether or not + that implementation started from ORT-provided utility code. - Keep cross-repository integration reproducible through pinned versions, compatibility checks, and CI. - Preserve existing tested behavior and supported consumers throughout the move. +## Terminology + +| Term | Meaning | +| --- | --- | +| Plugin EP API | The public ORT API used to implement a plugin EP: `OrtApi`, `OrtEpApi`, `OrtEpFactory`, and `OrtEp` | +| Staging root | `plugin-ep-webgpu/`, the in-tree directory the provider is consolidated into before the move | +| Conformance suite | The portable operator cases together with their execution and result semantics | +| Conformance kit | The versioned artifact published with an ORT release, containing cases, schemas, and a runner | +| Case archive | The platform-neutral conformance cases and schemas within a conformance kit | +| Provider profile | An EP's declared support surface, options, exclusions, and comparison overrides, supplied when running the conformance suite | + ## Workstreams The effort is divided into four workstreams: @@ -42,6 +58,24 @@ The effort is divided into four workstreams: The detailed reusable conformance-suite design is in [Execution Provider Operator Conformance Suite](test_ownership_and_conformance/ep_operator_conformance_design.md). +## Related work + +Two adjacent efforts target `onnxruntime-web` and are independent of this one: + +- [ORT Web JSEP to WebGPU EP migration](../onnxruntime_web_jsep_to_webgpu_ep_migration.md), with the user-facing + [JSEP deprecation notice](../../JSEP_Deprecation.md), replaces the deprecated JSEP TypeScript compute path with the + native WebGPU EP. +- [ORT Web WebGL backend removal](../onnxruntime_web_remove_webgl_backend.md) retires the WebGL backend. + +Neither effort gates this extraction, and this extraction does not gate them. `onnxruntime-web` includes the WebGPU +EP in some form regardless of when JSEP is removed, and both implementations register under the same `webgpu` backend +key, so changing which one a bundle ships requires no consumer source change. + +The efforts interact in one place: while both implementations coexist, the browser CI lanes are +implementation-specific even though the tests are not, so a passing default-bundle lane says nothing about the +provider being extracted. This is handled in +[Test ownership and operator conformance](test_ownership_and_conformance/test_ownership_and_conformance_workstream.md). + ## Workstream dependencies Most work proceeds concurrently, but one sequence determines the end date: @@ -58,9 +92,9 @@ graph LR Static plugin registration is a prerequisite for isolation, not merely an enabler. The `provider-isolation` workstream moves provider sources into the staging root rather than copying them, so once isolation completes there is exactly -one WebGPU implementation and it is adapter-free. The non-plugin static build that `onnxruntime-web` and the -`onnxruntime-webgpu` package ship from today must already be served through static plugin registration before the -adapter can be removed. Relocating the sources is not itself blocked; removing the adapter is. +one WebGPU implementation and it is adapter-free. The non-plugin static build that `onnxruntime-web` ships from today +must already be served through static plugin registration before the adapter can be removed. Relocating the sources +is not itself blocked; removing the adapter is. The remaining convergence points are not on the critical path: @@ -71,22 +105,24 @@ The remaining convergence points are not on the critical path: - The Node workstream consumes generic plugin loading from ORT and native WebGPU artifacts from the external provider. - The final repository copy depends on `plugin-ep-webgpu/` being independently buildable. -## Extraction launch matrix +## Consumer dispositions -The extraction launch matrix records existing consumers that require an explicit disposition before built-in code is -removed. Platform and architecture details remain to be inventoried in the individual workstreams. +Every consumer that receives WebGPU today needs a recorded disposition before the built-in implementation is removed +from ORT. Platform and architecture details remain to be inventoried in the individual workstreams. -| Consumer or package | Launch disposition | Extraction requirement | +| Consumer or package | Disposition | Extraction requirement | | --- | --- | --- | | `onnxruntime-web` | Consume a pinned external WebGPU source revision through static plugin registration | Required | | Python WebGPU plugin package | Move the existing optional plugin package and release pipeline | Required | | WebGPU NuGet plugin package | Move the existing optional plugin package and release pipeline | Required | | Node WebGPU support | Provide a tested replacement for the WebGPU implementation currently bundled in `onnxruntime-node`; final package naming is open | Required before bundled support is removed | -| Deprecated native packages with built-in WebGPU | Retire according to an explicit compatibility plan; do not make core packages depend on or bundle the plugin | Required disposition | -| New native hosts without current WebGPU support | Add later unless explicitly included in the launch matrix | Not an extraction prerequisite | +| `onnxruntime-webgpu` on PyPI | Publication has already stopped; do not resurrect it or convert it into a plugin-dependent package | Confirm whether any other retired package needs the same treatment | + +Hosts that do not ship WebGPU support today are outside this table. Adding one is a separate decision and is not an +extraction prerequisite. -The matrix should be updated as the current package and platform inventory is completed. Any removal from the launch -scope requires an explicit compatibility decision rather than silently dropping support. +The table should be updated as the current package and platform inventory is completed. Dropping a consumer requires +an explicit compatibility decision rather than silently removing support. ## Integrated delivery stages @@ -94,8 +130,9 @@ scope requires an explicit compatibility decision rather than silently dropping - Inventory private dependencies, build inputs, packages, consumers, platforms, and tests. - Record every current test's owner and intended destination. -- Establish correctness, package installation, WebAssembly size, and startup baselines. -- Confirm the launch matrix and platform coverage. +- Establish baselines for correctness, package installation, WebAssembly size, and inference latency. The later size + and performance gates are measured against these baselines. +- Confirm the consumer dispositions and platform coverage. ### Stage 2: Execute workstreams in parallel @@ -130,8 +167,8 @@ The source move is ready when: - Publish versioned native plugin artifacts and immutable source artifacts for static consumers. - Move Python and NuGet plugin releases to the WebGPU repository. - Pin the WebGPU revision used by `onnxruntime-web`. -- Execute the approved Node package transition. -- Retire deprecated built-in packages and remove obsolete implementation and packaging code from ORT. +- Execute the approved Node package transition, including any package retirement it implies. +- Remove obsolete WebGPU implementation and packaging code from ORT. ## Success criteria @@ -154,4 +191,4 @@ The source move is ready when: - What compatibility window should the WebGPU EP promise across ORT releases? - Should the Node transition keep the `onnxruntime-node` name or introduce a core-only package such as `onnxruntime-node-core`? -- How should coordinated ORT API and WebGPU provider changes be tested before either repository merges? +- May the WebGPU build-against ORT version reference a pre-release, to shorten the wait for a newly added EP API? From e8a42f53d10a6d87372ae47ee099cef0daddb12d Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Fri, 21 Aug 2026 17:10:27 -0700 Subject: [PATCH 007/107] Refine dependency scope and pipeline dispositions in provider isolation plan Splits the provider's non-public-API dependencies into three groups, adding plugin EP implementation utilities under include/onnxruntime/ep/. Those headers are offered for plugin EP implementations but are not shipped in the released package, so they are copied rather than consumed. Scopes the section to what the compiler and linker see, and moves the build-input inventory to the standalone build contract. Replaces the single disposition for CI lanes with three: lanes that build the provider statically into ORT are retired, onnxruntime-web keeps static linkage against the pinned external source, and provider-owned lanes move to the WebGPU repository. Records that ORT-root packaging scripts reference WebGPU independently of the plugin packages, and that the plugin-ep-webgpu/rel-* branch prefix disappears once release branches no longer share the ORT repository. Adds the browser-test lane to the cross-repository version policy, which the standalone build contract already pointed at it for. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- ...ion_and_repository_migration_workstream.md | 56 +++++++++++++------ 1 file changed, 39 insertions(+), 17 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md b/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md index 9e290cadbeca9..0cd07f14bff26 100644 --- a/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md +++ b/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md @@ -85,25 +85,31 @@ During isolation: The provider uses ORT-internal code deliberately. While the implementation and the runtime live in one repository, reusing ORT's kernel-authoring types and operator helpers avoids duplicating code that is already maintained next to -it. That trade stops paying once the provider ships from its own repository, and unwinding it is a large part of what -the move is for. The coupling is not accidental, and removing it is not a correction. +it. Moving the provider removes the condition that made that trade worthwhile: code it cannot reach from another +repository has to be replaced or copied before the subtree builds on its own. Unwinding the coupling is a +precondition of the move. -The provider's dependencies on ORT source fall into two groups with different dispositions: +The dividing line is the public ORT API. Everything else the provider depends on has to be addressed, whether it is +private implementation code or a utility ORT offers but does not ship. Those dependencies fall into three groups with +different dispositions: - **Framework surface** — the kernel-authoring types the provider is written against. Replacing these is the kernel-authoring foundation work in the code isolation package, not a copy decision. - **Operator helpers** — parameter parsing, shape math, and similar utilities shared with the CPU and CUDA EPs today only because everything lives in one repository. Copying these is the intended outcome, and the provider owns the correctness of its copies afterwards. +- **Plugin EP implementation utilities** — `include/onnxruntime/ep/api.h`, `common.h`, and + `get_capability_utils.h`. ORT offers these for plugin EP implementations to use, and they depend only on the public + C API, gsl, and the standard library. They are not shipped in the released package, so they are not public API and + the provider copies them like anything else in this section. The adapter headers under + `include/onnxruntime/ep/adapter/` are a separate tier that the extraction retires rather than copies. -The inventory is broader than source dependencies. Create a machine-reviewable inventory of: +Create a machine-reviewable inventory of: - Private ORT headers included by provider sources. - Private ORT libraries in provider link interfaces. - Source files compiled into WebGPU targets from outside the provider root. -- Build variables and generated files supplied by ORT. - Test-only dependencies on ORT internals. -- Platform and package scripts that assume an in-tree provider. Classify each dependency: @@ -128,6 +134,9 @@ with the ORT implementation. The isolated subtree should build against the public ORT headers and an installed or pinned ORT package, without an ORT source checkout. +Inventory the build variables and generated files ORT's build currently supplies to the provider, since each one is +either reproduced by the subtree build or becomes an input it must be given. + The subtree build should produce: - A native shared plugin library. @@ -149,21 +158,32 @@ Web with the provider, so the WebGPU repository builds ORT Web from a pinned ORT adjacent-checkout override. That revision is selected by the cross-repository version policy below. ORT separately validates its pinned WebGPU revision as part of ORT Web release gating. -## Packaging and pipeline migration +## Pipeline and packaging migration -Python and NuGet plugin packaging sources already live under `plugin-ep-webgpu/`. What remains outside the staging -root is the packaging and CI pipeline definitions that drive them, under `tools/ci_build/` and `.github/workflows/`. -This workstream relocates those and rewires them to invoke the standalone build, but does not redesign the packaging -scripts themselves. +Python and NuGet plugin packaging sources already live under `plugin-ep-webgpu/`. The plugin build, test, and +packaging pipeline definitions are still outside the staging root, split between +`tools/ci_build/github/azure-pipelines/` and `.github/workflows/`. This workstream relocates those and rewires them +to invoke the standalone build, but does not redesign the packaging scripts themselves. The pipelines invoke `tools/ci_build/build.py --use_webgpu shared_lib` and consume artifacts from ORT's build output locations, so they break the moment the standalone build replaces the in-tree one. The rewiring therefore lands with the standalone build rather than after it, and no temporary shim over `build.py` is maintained. Relocating the pipeline files is mechanical and happens earlier, with the rest of the staging-root move. -ORT workflows that merely trigger on `plugin-ep-webgpu/rel-*` branches stay in ORT and need only their trigger -configuration updated. Shared assets such as the WebGPU shader-key validation action are consumed by both moving and -staying lanes, so their ownership is decided individually. +ORT-root packaging scripts reference WebGPU independently of the plugin packages, such as `setup.py` selecting the +retired `onnxruntime-webgpu` package name from a `--use_webgpu` flag. Inventory these alongside the pipelines. They +belong to the retired built-in package rather than to the plugin, so they are removed with it rather than relocated. + +Most ORT CI lanes live under `.github/workflows/`, and the WebGPU ones do not share a single disposition. Lanes that +build the provider statically into ORT validate a configuration that stops existing, so they are retired rather than +rewired. `onnxruntime-web` is the exception: it keeps static linkage, against the pinned external source, along with +the WebAssembly build and browser-test lanes that serve it. Provider-owned concerns move to the WebGPU repository, +including external-Dawn validation, WGSL shader-key validation together with its action at +`.github/actions/webgpu-validate-shader-key`, and the plugin shared-library build. + +The `plugin-ep-webgpu/rel-*` branch prefix exists only because WebGPU release branches share the ORT repository. +Pipelines that stay in ORT drop that trigger entirely, and pipelines that move use ordinary release branches in the +WebGPU repository. The packaging model is: @@ -251,18 +271,20 @@ other. Only one lane floats, and it never blocks a merge: | --- | --- | --- | | WebGPU build | A released ORT version providing every EP API feature the provider references | Yes | | WebGPU minimum-version validation | The declared `MIN_ONNXRUNTIME_VERSION` runtime | Yes | +| WebGPU browser tests | ORT Web built from the same released ORT revision as the build lane, consumed as source rather than as a package | Yes | | WebGPU integration | ORT main | No | | ORT | Its pinned WebGPU revision | Yes | -Both blocking WebGPU lanes target immutable released ORT artifacts, so neither can wait on an ORT revision that does -not yet exist. ORT advances its WebGPU pin on its own schedule and declines the update when it fails. +The blocking WebGPU lanes all target immutable released ORT artifacts — a package for the build and minimum-version +lanes, a source revision for browser tests — so none of them can wait on an ORT revision that does not yet exist. ORT +advances its WebGPU pin on its own schedule and declines the update when it fails. Two ORT versions matter here, and they are not the same number: - The **build-against version** must declare every EP API the provider references, including calls reached only through a runtime version gate, because a gated call still needs its declaration to compile. - The **runtime floor**, `MIN_ONNXRUNTIME_VERSION`, is the oldest runtime the provider loads against. It can be - lower, because newer calls are gated at runtime through `CurrentOrtApiVersion()`. + lower, because newer calls are gated on the ORT API version detected at runtime. They coincide only while nothing is gated above the floor. A build against newer headers cannot detect a mis-gated call, so the floor has to be exercised by running against it. That is the minimum-version validation package. From 800ad881efc0145f71591c593903ac65d18b4390 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Mon, 24 Aug 2026 11:20:59 -0700 Subject: [PATCH 008/107] Resolve internal contradictions found in PR review Scopes CPU-fallback disabling to non-CPU targets. The conformance design required running cases against CPU as a reference while unconditionally disabling CPU fallback, which ORT rejects whenever nodes are assigned to the CPU EP. Makes the static-runner SDK conditional in the design section, Phase 3, and the MVP deliverables, since whether a native static-linkage runner is needed at all is still an open question. Qualifies the isolation milestone: every build input is either inside the staging root or a declared external dependency. The previous wording forbade any input resolving outside the root, which the standalone build contract's own ORT package and pinned third-party dependencies could not satisfy. Reframes the cross-repository version policy around avoiding a circular dependency rather than claiming neither side can be blocked, and names the wait for an ORT release as a scheduling cost rather than a deadlock. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- ...isolation_and_repository_migration_workstream.md | 13 ++++++++----- .../ep_operator_conformance_design.md | 12 +++++++----- .../test_ownership_and_conformance_workstream.md | 3 ++- 3 files changed, 17 insertions(+), 11 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md b/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md index 0cd07f14bff26..fa043d3dab4f0 100644 --- a/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md +++ b/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md @@ -264,8 +264,8 @@ The dependency inventory should compare existing ORT mechanisms before selecting ### Cross-repository version policy -Each repository pins the other, so the pins must be arranged so that neither side can be blocked waiting for the -other. Only one lane floats, and it never blocks a merge: +Each repository pins the other, so the pins must be arranged so that the dependency does not become circular. Only +one lane floats, and it never blocks a merge: | Lane | Built or run against | Blocking | | --- | --- | --- | @@ -298,7 +298,9 @@ public plugin EP API. The exception is a provider dependency on unspecified beha repository. Without a stated owner the lane goes permanently red and stops being read. Adopting a newly added EP API therefore requires an ORT release carrying it before the provider can build against it. -This does not force the runtime floor upward, since the new call can be gated. +This does not force the runtime floor upward, since the new call can be gated. The wait is a scheduling cost rather +than a deadlock: both repositories keep landing changes while it elapses, and the open question about consuming an +ORT pre-release exists to shorten it. ## Work packages @@ -361,8 +363,9 @@ Sequencing: ### Isolation milestone The staging root is ready for transfer when a clean copy of it builds, tests, and packages everything listed in -Desired end state, with no provider build input resolving outside the root and no private ORT headers or libraries in -the link interface. +Desired end state, with every build input either inside the root or a declared external dependency, and no private +ORT headers or libraries in the link interface. The ORT package and the pinned third-party dependencies are declared +external inputs, not exceptions to the milestone. ### End state diff --git a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md index 304d1bf664ebb..39c64437a9294 100644 --- a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md +++ b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md @@ -166,7 +166,7 @@ For an ordinary operator conformance case, the runner should: 1. Load or register the requested plugin EP and select a device. 2. Apply the requested EP options. -3. Create a session with CPU fallback disabled. +3. Create a session with CPU fallback disabled, unless the CPU EP is itself the target. 4. Load the generated or packaged model. 5. Require the target EP to accept the nodes specified by the case. 6. Run every input dataset. @@ -174,7 +174,8 @@ For an ordinary operator conformance case, the runner should: 8. Record diagnostics without changing the defined outcome. Disabling CPU fallback is essential. A correct result produced by the CPU EP does not demonstrate conformance of the -target EP. +target EP. The exception is the CPU reference run, where the CPU EP is the target: there is nothing to fall back +from, and ORT rejects a session that disables CPU fallback while nodes are assigned to the CPU EP. Cases that intentionally test partial graph assignment must state their assignment requirements explicitly. They should be classified separately from single-operator correctness cases. @@ -267,8 +268,9 @@ device discovery, session creation, and execution APIs. ## Static plugin usage -A prebuilt executable cannot discover a statically linked plugin. ORT should therefore also publish a small runner -SDK or CMake target that allows an EP repository to supply static factory registration: +A prebuilt executable cannot discover a statically linked plugin, so validating static linkage requires a runner the +EP repository builds itself. Whether that runner is needed is an open question below. If it is adopted, ORT would +also publish a small runner SDK or CMake target that allows an EP repository to supply static factory registration: ```cmake find_package(onnxruntime_ep_conformance CONFIG REQUIRED) @@ -330,7 +332,7 @@ The report should make newly unsupported or newly excluded cases easy to detect - Publish native runners and platform-neutral case archives with ORT releases. - Add a nightly kit for ORT `main`. -- Add a static-runner SDK or CMake package. +- Add a static-runner SDK or CMake package, if a native static-linkage runner is adopted. - Integrate the conformance kit into an external plugin EP repository. ### Phase 4: Expand coverage and hosts diff --git a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md index 8ecd25983ee3b..b263b3bf1e1c9 100644 --- a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md +++ b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md @@ -105,7 +105,8 @@ The MVP should provide: - A shared conformance-runner core for case loading, execution, comparison, and reporting. - A prebuilt native executable that uses the core and loads dynamic plugin EP libraries. -- A small SDK or CMake target that uses the same core and lets an external EP link its static factory registration. +- A small SDK or CMake target that uses the same core and lets an external EP link its static factory registration, + if a native static-linkage runner is adopted. - CPU fallback prevention or reliable assignment verification. - Case, provider-profile, and report schemas. - Structured `PASS`, `UNSUPPORTED`, `FAIL`, `EXCLUDED`, and `NOT_RUN` results, with semantics defined in From 275467422973ec777d4f44e3b78b01ddba91151b Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Tue, 25 Aug 2026 16:58:05 -0700 Subject: [PATCH 009/107] Support registering statically linked plugin EPs, and build WebGPU EP that way Adds a generic mechanism for plugin EPs that are linked into the ORT binary rather than loaded from a shared library, and uses it for the WebGPU EP via a new --use_webgpu static_plugin build configuration. This lets one set of EP sources (core/providers/webgpu/ep) serve both a dynamically loaded plugin library and a statically linked build. The latter is what ORT Web needs, since Emscripten cannot load shared library modules. Core: - EpLibraryStaticPlugin, an EpLibrary that takes the factory entry points directly instead of resolving them from a loaded library. The factory lifecycle it shares with EpLibraryPlugin is factored into ep_library_plugin_utils. - ep_static_plugins.cc, a link-time registry of statically linked plugin EPs. - Environment::CreateAndRegisterStaticPluginEps(). The registration runs from OrtEnv::GetOrCreateInstance() rather than from Environment::Initialize(), after the singleton is published and its reference taken. Registering an EP enumerates its devices, and the WebGPU factory queries the environment while doing so, which self-deadlocks if the environment is not yet published. OrtEnv::m_ becomes recursive to allow that re-entry. WebGPU EP: - Entry point names are macro-parameterized so the statically linked build can emit prefixed symbols. The shared library build must keep the unprefixed names that EpLibraryPlugin::Load looks up. - Protobuf shutdown is compiled out unless the EP owns the process globals, which is only true for the shared library module. - ORT_PLUGIN_EP_STATICALLY_LINKED disables manual C++ API initialization. The MSVC detect_mismatch guard on ORT_API_MANUAL_INIT requires whole-binary agreement, so forcing it on in ep/api.h fails to link once EP objects are part of the ORT binary. Initialization is unnecessary there anyway: the EP and ORT are the same binary and therefore the same API version. Tests: - The plugin EP test infrastructure treats the EP library path as optional, since ORT core registers statically linked plugin EPs itself. - Virtual devices are requested through the allow_virtual_devices environment configuration entry rather than a ".virtual" registration name suffix, which is unavailable when ORT core chooses the registration name. - The WebGPU plugin EP test definitions move into a shared cmake function and are applied to onnxruntime_provider_test as well as onnxruntime_test_all. The operator tests live in the former, so the plugin path had no operator test coverage. Design notes are in docs/design/webgpu_ep_extraction. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- cmake/CMakeLists.txt | 20 ++ cmake/onnxruntime.cmake | 2 +- cmake/onnxruntime_providers_webgpu.cmake | 88 ++++-- cmake/onnxruntime_python.cmake | 2 +- cmake/onnxruntime_unittests.cmake | 48 ++- ...boundary_and_web_integration_workstream.md | 5 +- .../static_plugin_ep_registration_design.md | 293 ++++++++++++++++++ .../webgpu_ep_extraction.md | 3 + .../onnxruntime/core/session/environment.h | 14 + include/onnxruntime/ep/api.h | 16 +- onnxruntime/core/providers/webgpu/ep/api.cc | 41 ++- onnxruntime/core/session/environment.cc | 12 + onnxruntime/core/session/ort_env.cc | 30 +- onnxruntime/core/session/ort_env.h | 4 +- .../session/plugin_ep/ep_library_plugin.cc | 46 +-- .../plugin_ep/ep_library_plugin_utils.cc | 67 ++++ .../plugin_ep/ep_library_plugin_utils.h | 38 +++ .../plugin_ep/ep_library_static_plugin.cc | 45 +++ .../plugin_ep/ep_library_static_plugin.h | 55 ++++ .../session/plugin_ep/ep_static_plugins.cc | 38 +++ .../session/plugin_ep/ep_static_plugins.h | 17 + .../test/framework/inference_session_test.cc | 20 +- onnxruntime/test/unittest_main/test_main.cc | 41 +++ .../unittest_util/test_dynamic_plugin_ep.cc | 13 +- .../unittest_util/test_dynamic_plugin_ep.h | 4 + tools/ci_build/build.py | 3 + tools/ci_build/build_args.py | 7 +- 27 files changed, 859 insertions(+), 113 deletions(-) create mode 100644 docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md create mode 100644 onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.cc create mode 100644 onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.h create mode 100644 onnxruntime/core/session/plugin_ep/ep_library_static_plugin.cc create mode 100644 onnxruntime/core/session/plugin_ep/ep_library_static_plugin.h create mode 100644 onnxruntime/core/session/plugin_ep/ep_static_plugins.cc create mode 100644 onnxruntime/core/session/plugin_ep/ep_static_plugins.h diff --git a/cmake/CMakeLists.txt b/cmake/CMakeLists.txt index 2572b1657d1cf..5ecf406541a3c 100644 --- a/cmake/CMakeLists.txt +++ b/cmake/CMakeLists.txt @@ -176,6 +176,7 @@ option(onnxruntime_USE_XNNPACK "Build with XNNPACK support. Provides an alternat option(onnxruntime_USE_WEBNN "Build with WebNN support. Enable hardware acceleration in web browsers." OFF) option(onnxruntime_USE_WEBGPU "Build with WebGPU support. Enable WebGPU via C/C++ interface." OFF) option(onnxruntime_USE_EP_API_ADAPTERS "Build EP (e.g. WebGPU) as a plugin EP shared library using EP API adapters" OFF) +option(onnxruntime_WEBGPU_STATIC_PLUGIN "Build the WebGPU plugin EP into the ORT binary instead of a separate shared library. Requires onnxruntime_USE_EP_API_ADAPTERS." OFF) option(onnxruntime_USE_EXTERNAL_DAWN "Build with treating Dawn as external dependency. Will not link Dawn at build time." OFF) option(onnxruntime_CUSTOM_DAWN_SRC_PATH "Path to custom Dawn src dir.") option(onnxruntime_BUILD_DAWN_SHARED_LIBRARY "Build Dawn as a shared library" OFF) @@ -440,6 +441,25 @@ if(onnxruntime_USE_EP_API_ADAPTERS) add_compile_definitions(ORT_USE_EP_API_ADAPTERS=1) endif() +# The WebGPU plugin EP can either be a separate shared library or be linked into the ORT binary. In the latter case +# ORT core registers it at OrtEnv creation time instead of the application loading it by path. +if(onnxruntime_WEBGPU_STATIC_PLUGIN) + if(NOT onnxruntime_USE_WEBGPU) + message(FATAL_ERROR "onnxruntime_WEBGPU_STATIC_PLUGIN=ON requires onnxruntime_USE_WEBGPU=ON.") + endif() + if(NOT onnxruntime_USE_EP_API_ADAPTERS) + message(FATAL_ERROR "onnxruntime_WEBGPU_STATIC_PLUGIN=ON requires onnxruntime_USE_EP_API_ADAPTERS=ON.") + endif() + add_compile_definitions(ORT_WEBGPU_STATIC_PLUGIN=1) +endif() + +# True when the WebGPU EP is compiled into the ORT binary rather than built as a separate shared library. +if(onnxruntime_USE_WEBGPU AND (NOT onnxruntime_USE_EP_API_ADAPTERS OR onnxruntime_WEBGPU_STATIC_PLUGIN)) + set(onnxruntime_WEBGPU_LINKED_INTO_HOST ON) +else() + set(onnxruntime_WEBGPU_LINKED_INTO_HOST OFF) +endif() + # General C# properties if (onnxruntime_BUILD_CSHARP) check_language(CSharp) diff --git a/cmake/onnxruntime.cmake b/cmake/onnxruntime.cmake index a6cc69d751eef..87d5ab38c08ee 100644 --- a/cmake/onnxruntime.cmake +++ b/cmake/onnxruntime.cmake @@ -257,7 +257,7 @@ if (onnxruntime_BUILD_QNN_EP_STATIC_LIB) list(APPEND onnxruntime_INTERNAL_PROVIDER_LIBRARIES onnxruntime_providers_qnn) endif() -if (onnxruntime_USE_WEBGPU AND NOT onnxruntime_USE_EP_API_ADAPTERS) +if (onnxruntime_WEBGPU_LINKED_INTO_HOST) list(APPEND onnxruntime_INTERNAL_PROVIDER_LIBRARIES onnxruntime_providers_webgpu) endif() diff --git a/cmake/onnxruntime_providers_webgpu.cmake b/cmake/onnxruntime_providers_webgpu.cmake index 85d1404bb4e50..5b6e45e8f57e0 100644 --- a/cmake/onnxruntime_providers_webgpu.cmake +++ b/cmake/onnxruntime_providers_webgpu.cmake @@ -21,7 +21,7 @@ if(NOT onnxruntime_USE_EP_API_ADAPTERS) # - # Build WebGPU EP as a static library + # Build WebGPU EP as an internal (non-plugin) static library # # For static library build, exclude the 'ep' folder @@ -35,6 +35,27 @@ onnxruntime_add_static_library(onnxruntime_providers_webgpu ${onnxruntime_providers_webgpu_cc_srcs}) onnxruntime_add_include_to_target(onnxruntime_providers_webgpu onnxruntime_common onnx onnx_proto flatbuffers::flatbuffers Boost::mp11 safeint_interface) + elseif(onnxruntime_WEBGPU_STATIC_PLUGIN) + # + # Build WebGPU EP as a plugin EP that is statically linked into the ORT binary. + # + # This uses the same plugin EP sources ('ep' folder) and EP API adapters as the shared library build below. + # The difference is only in how the EP reaches the application: ORT core registers it at OrtEnv creation time + # instead of the application calling RegisterExecutionProviderLibrary() with a library path. + # + source_group(TREE ${ONNXRUNTIME_ROOT} FILES ${onnxruntime_providers_webgpu_cc_srcs}) + onnxruntime_add_static_library(onnxruntime_providers_webgpu ${onnxruntime_providers_webgpu_cc_srcs}) + onnxruntime_add_include_to_target(onnxruntime_providers_webgpu + onnxruntime_common onnx onnx_proto flatbuffers::flatbuffers Boost::mp11 safeint_interface) + + # Prefix the plugin EP entry points so that multiple statically linked plugin EPs can coexist in one binary. + # ORT core declares the prefixed names in onnxruntime/core/session/plugin_ep/ep_static_plugins.cc. + # + # ORT_PLUGIN_EP_STATICALLY_LINKED tells the shared plugin EP headers that OrtGetApiBase() is available in-process, + # so the C++ API must not be built with ORT_API_MANUAL_INIT (the rest of the binary is not). + target_compile_definitions(onnxruntime_providers_webgpu PRIVATE + ORT_PLUGIN_EP_ENTRY_POINT_PREFIX=WebGpu_ + ORT_PLUGIN_EP_STATICALLY_LINKED=1) else() # # Build WebGPU EP as a shared library @@ -73,26 +94,9 @@ add_definitions("-DONNX_NAMESPACE=onnx") add_definitions("-DONNX_USE_LITE_PROTO=1") - # Default plugin EP version to ORT_VERSION with "-dev" suffix if not explicitly provided. - if(NOT DEFINED onnxruntime_PLUGIN_EP_VERSION) - set(onnxruntime_PLUGIN_EP_VERSION "${ORT_VERSION}-dev") - endif() - - # Set preprocessor definition for plugin EP version - target_compile_definitions(onnxruntime_providers_webgpu PRIVATE - ORT_PLUGIN_EP_VERSION="${onnxruntime_PLUGIN_EP_VERSION}") - - # Bake the minimum compatible ORT version (the single source of truth lives in - # plugin-ep-webgpu/MIN_ONNXRUNTIME_VERSION) into the EP DLL so it can be enforced at runtime. - # Format is strict "MAJOR.MINOR.PATCH". - set(_ORT_PLUGIN_EP_WEBGPU_MIN_ORT_VERSION_FILE "${REPO_ROOT}/plugin-ep-webgpu/MIN_ONNXRUNTIME_VERSION") - file(STRINGS "${_ORT_PLUGIN_EP_WEBGPU_MIN_ORT_VERSION_FILE}" _ORT_PLUGIN_EP_WEBGPU_MIN_ORT_VERSION LIMIT_COUNT 1) - if(NOT _ORT_PLUGIN_EP_WEBGPU_MIN_ORT_VERSION) - message(FATAL_ERROR "WebGPU plugin EP minimum ORT version file is missing or empty: " - "${_ORT_PLUGIN_EP_WEBGPU_MIN_ORT_VERSION_FILE}") - endif() - target_compile_definitions(onnxruntime_providers_webgpu PRIVATE - ORT_PLUGIN_EP_MIN_ORT_VERSION="${_ORT_PLUGIN_EP_WEBGPU_MIN_ORT_VERSION}") + # This EP is its own module, so it owns the lifetime of the process-global state it initializes + # (e.g. the protobuf library) and shuts it down when the factory is released. + target_compile_definitions(onnxruntime_providers_webgpu PRIVATE ORT_PLUGIN_EP_OWNS_PROCESS_GLOBALS=1) # Set preprocessor definitions used in onnxruntime_providers_webgpu.rc if(WIN32) @@ -127,11 +131,38 @@ message(FATAL_ERROR "WebGPU EP shared library build does not support build cache. Please disable build cache or use static library build.") endif() if (CMAKE_SYSTEM_NAME STREQUAL "Emscripten") - message(FATAL_ERROR "WebGPU EP shared library build is not supported on Emscripten. Please use static library build.") + message(FATAL_ERROR "WebGPU EP shared library build is not supported on Emscripten. Please use '--use_webgpu static_plugin'.") + endif() + endif() + + if (onnxruntime_USE_EP_API_ADAPTERS) + # + # Settings common to both plugin EP builds (shared library and statically linked). + # + + # Default plugin EP version to ORT_VERSION with "-dev" suffix if not explicitly provided. + if(NOT DEFINED onnxruntime_PLUGIN_EP_VERSION) + set(onnxruntime_PLUGIN_EP_VERSION "${ORT_VERSION}-dev") + endif() + + # Set preprocessor definition for plugin EP version + target_compile_definitions(onnxruntime_providers_webgpu PRIVATE + ORT_PLUGIN_EP_VERSION="${onnxruntime_PLUGIN_EP_VERSION}") + + # Bake the minimum compatible ORT version (the single source of truth lives in + # plugin-ep-webgpu/MIN_ONNXRUNTIME_VERSION) into the EP so it can be enforced at runtime. + # Format is strict "MAJOR.MINOR.PATCH". + set(_ORT_PLUGIN_EP_WEBGPU_MIN_ORT_VERSION_FILE "${REPO_ROOT}/plugin-ep-webgpu/MIN_ONNXRUNTIME_VERSION") + file(STRINGS "${_ORT_PLUGIN_EP_WEBGPU_MIN_ORT_VERSION_FILE}" _ORT_PLUGIN_EP_WEBGPU_MIN_ORT_VERSION LIMIT_COUNT 1) + if(NOT _ORT_PLUGIN_EP_WEBGPU_MIN_ORT_VERSION) + message(FATAL_ERROR "WebGPU plugin EP minimum ORT version file is missing or empty: " + "${_ORT_PLUGIN_EP_WEBGPU_MIN_ORT_VERSION_FILE}") endif() + target_compile_definitions(onnxruntime_providers_webgpu PRIVATE + ORT_PLUGIN_EP_MIN_ORT_VERSION="${_ORT_PLUGIN_EP_WEBGPU_MIN_ORT_VERSION}") - # Configure precompiled headers for shared library build - # PCH ensures ep/adapters.h is included first and improves compilation speed + # Configure precompiled headers for the plugin EP build. + # PCH ensures ep/adapters.h is included first and improves compilation speed. target_precompile_headers(onnxruntime_providers_webgpu PRIVATE "${REPO_ROOT}/include/onnxruntime/ep/adapters.h" ) @@ -255,13 +286,14 @@ endif() endif() - # In the plugin/adapter build (onnxruntime_USE_EP_API_ADAPTERS) the WebGPU EP is its own DLL - # (onnxruntime_providers_webgpu.dll) rather than being statically linked into onnxruntime.dll, so the + # In the plugin EP shared library build the WebGPU EP is its own DLL + # (onnxruntime_providers_webgpu.dll) rather than being linked into onnxruntime.dll, so the # delay-load flags accumulated above (notably /DELAYLOAD:user32.dll) must be applied to THIS target. # Without it the DLL keeps a static import on user32.dll and fails to load in a Win32k-lockdown sandbox # (e.g. Chromium's WebNN compiler process): 'depends on "USER32.dll" which is missing' (Error 1359). - # In the static build these same flags are applied to the onnxruntime target in onnxruntime.cmake. - if (WIN32 AND onnxruntime_USE_EP_API_ADAPTERS AND onnxruntime_DELAYLOAD_FLAGS) + # When the EP is linked into the ORT binary these same flags are applied to the onnxruntime target in + # onnxruntime.cmake. + if (WIN32 AND NOT onnxruntime_WEBGPU_LINKED_INTO_HOST AND onnxruntime_DELAYLOAD_FLAGS) target_link_options(onnxruntime_providers_webgpu PRIVATE ${onnxruntime_DELAYLOAD_FLAGS}) target_link_libraries(onnxruntime_providers_webgpu PRIVATE delayimp.lib) endif() diff --git a/cmake/onnxruntime_python.cmake b/cmake/onnxruntime_python.cmake index 80fbbdf0e7260..20c0830547d96 100644 --- a/cmake/onnxruntime_python.cmake +++ b/cmake/onnxruntime_python.cmake @@ -213,7 +213,7 @@ set(onnxruntime_pybind11_state_static_providers if(onnxruntime_BUILD_QNN_EP_STATIC_LIB) list(APPEND onnxruntime_pybind11_state_static_providers PRIVATE onnxruntime_providers_qnn) endif() -if(onnxruntime_USE_WEBGPU AND NOT onnxruntime_USE_EP_API_ADAPTERS) +if(onnxruntime_WEBGPU_LINKED_INTO_HOST) list(APPEND onnxruntime_pybind11_state_static_providers PRIVATE onnxruntime_providers_webgpu) endif() if(WIN32) diff --git a/cmake/onnxruntime_unittests.cmake b/cmake/onnxruntime_unittests.cmake index e500645f22377..4ce03b02fd77a 100644 --- a/cmake/onnxruntime_unittests.cmake +++ b/cmake/onnxruntime_unittests.cmake @@ -719,7 +719,7 @@ set(ONNXRUNTIME_TEST_STATIC_PROVIDER_LIBS if (onnxruntime_BUILD_QNN_EP_STATIC_LIB) list(APPEND ONNXRUNTIME_TEST_STATIC_PROVIDER_LIBS onnxruntime_providers_qnn) endif() -if (onnxruntime_USE_WEBGPU AND NOT onnxruntime_USE_EP_API_ADAPTERS) +if (onnxruntime_WEBGPU_LINKED_INTO_HOST) list(APPEND ONNXRUNTIME_TEST_STATIC_PROVIDER_LIBS onnxruntime_providers_webgpu) endif() @@ -1185,7 +1185,11 @@ endfunction() # Set environment variables for plugin EP tests when run via CTest. function(onnxruntime_set_plugin_ep_test_environment target) - if(onnxruntime_USE_WEBGPU AND onnxruntime_USE_EP_API_ADAPTERS) + if(onnxruntime_WEBGPU_STATIC_PLUGIN) + # The WebGPU plugin EP is linked into the test binary and registered by ORT core, so there is no + # library to register by path here. onnxruntime_set_webgpu_plugin_ep_test_definitions() below gives + # test_main.cc an equivalent compiled-in default config. + elseif(onnxruntime_USE_WEBGPU AND onnxruntime_USE_EP_API_ADAPTERS) set(ORT_PLUGIN_EP_JSON_CONFIG "{\"ep_library_registration_name\": \"WebGPU_PluginEP\", \"ep_library_path\": \"$\", \"selected_ep_name\": \"WebGpuExecutionProvider\"}") set_tests_properties(${target} PROPERTIES ENVIRONMENT "ORT_UNIT_TEST_MAIN_DYNAMIC_PLUGIN_EP_CONFIG_JSON=${ORT_PLUGIN_EP_JSON_CONFIG}" @@ -1195,6 +1199,33 @@ function(onnxruntime_set_plugin_ep_test_environment target) endif() endfunction() +# Route the WebGPU EP through the dynamic plugin EP infrastructure in plugin builds +# (--use_webgpu shared_lib or --use_webgpu static_plugin). +# Without initializing the infra in test_main.cc, WebGpuExecutionProviderWithOptions() (default_providers.cc, +# adapters branch) returns null and every WebGPU test skips itself, leaving the plugin path with no test coverage. +# This must be applied to every test target that links test_main.cc and runs WebGPU tests. +function(onnxruntime_set_webgpu_plugin_ep_test_definitions target) + if(NOT (onnxruntime_USE_WEBGPU AND onnxruntime_USE_EP_API_ADAPTERS)) + return() + endif() + + target_compile_definitions(${target} PRIVATE + ORT_UNIT_TEST_ENABLE_DYNAMIC_PLUGIN_EP_USAGE + ORT_UNIT_TEST_HAS_WEBGPU_PLUGIN_EP=1) + + if (onnxruntime_WEBGPU_STATIC_PLUGIN) + # The plugin EP is linked into the test binary and registered by ORT core during environment creation, + # so there is no library path to dlopen and no separate build-order dependency needed. + target_compile_definitions(${target} PRIVATE ORT_UNIT_TEST_HAS_WEBGPU_STATIC_PLUGIN_EP=1) + else() + target_compile_definitions(${target} PRIVATE + ORT_UNIT_TEST_WEBGPU_PLUGIN_EP_LIBRARY_PATH="$") + # The plugin EP DLL is dlopen'd at test-run time (not linked), so add an explicit build-order + # dependency to ensure it (and its co-located dawn/dxcompiler DLLs) exist before the tests run. + add_dependencies(${target} onnxruntime_providers_webgpu) + endif() +endfunction() + function(onnxruntime_apply_emscripten_test_link_settings target) if (CMAKE_SYSTEM_NAME STREQUAL "Emscripten") set_target_properties(${target} PROPERTIES LINK_DEPENDS ${TEST_SRC_DIR}/wasm/onnxruntime_test_adapter.js) @@ -1243,17 +1274,7 @@ if (onnxruntime_USE_CUDA AND onnxruntime_BUILD_CUDA_EP_AS_PLUGIN) endif() if (onnxruntime_USE_WEBGPU AND onnxruntime_USE_EP_API_ADAPTERS) - # Route the WebGPU EP through the dynamic plugin EP infrastructure in plugin builds - # (--use_webgpu shared_lib). Same rationale as the CUDA-as-plugin block above: without initializing the - # infra in test_main.cc, WebGpuExecutionProviderWithOptions() (default_providers.cc, adapters branch) - # returns null and every WebGPU test skips itself, leaving the plugin path with no test coverage. - target_compile_definitions(onnxruntime_test_all PRIVATE - ORT_UNIT_TEST_ENABLE_DYNAMIC_PLUGIN_EP_USAGE - ORT_UNIT_TEST_WEBGPU_PLUGIN_EP_LIBRARY_PATH="$" - ORT_UNIT_TEST_HAS_WEBGPU_PLUGIN_EP=1) - # The plugin EP DLL is dlopen'd at test-run time (not linked), so add an explicit build-order - # dependency to ensure it (and its co-located dawn/dxcompiler DLLs) exist before the tests run. - add_dependencies(onnxruntime_test_all onnxruntime_providers_webgpu) + onnxruntime_set_webgpu_plugin_ep_test_definitions(onnxruntime_test_all) endif() if (MSVC) @@ -1479,6 +1500,7 @@ block() # enable dynamic plugin EP usage target_compile_definitions(onnxruntime_provider_test PRIVATE ORT_UNIT_TEST_ENABLE_DYNAMIC_PLUGIN_EP_USAGE) + onnxruntime_set_webgpu_plugin_ep_test_definitions(onnxruntime_provider_test) onnxruntime_apply_emscripten_test_link_settings(onnxruntime_provider_test) if (IOS) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md index 2ff7f53894775..8dcbba70d3615 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md @@ -46,6 +46,8 @@ The design must address: The facility must be generic and validated with at least one non-WebGPU test plugin where practical. +The detailed design is in [Static Plugin EP Registration](static_plugin_ep_registration_design.md). + ## Process-global ownership and teardown Static registration removes the dynamic-library unload boundary. The current WebGPU plugin cleanup path cannot be @@ -176,5 +178,6 @@ The first five packages can proceed largely in parallel. Legacy-path removal wai ## Open questions -- Should static factories be registered before environment creation or through environment construction options? +- Resolved: neither. ORT core registers static factories during `OrtEnv` creation, after the environment is + constructed and published. See [Static Plugin EP Registration](static_plugin_ep_registration_design.md). - What is the stable representation of JavaScript-owned WebGPU objects at the C API boundary? diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md new file mode 100644 index 0000000000000..b2e42d668aedc --- /dev/null +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -0,0 +1,293 @@ +# Static Plugin EP Registration + +Status: Detailed design supporting +[Plugin Boundary and Web/Wasm Integration](plugin_boundary_and_web_integration_workstream.md). + +## Purpose + +Define how a plugin execution provider (EP) that is linked into the host binary is registered with ONNX Runtime, so +that statically linked and dynamically loaded plugin EPs share one provider implementation and one public API +boundary. + +WebGPU is the first consumer. The immediate deliverable is to move the statically linked WebGPU build off +`EpLibraryInternal` and onto the plugin EP path. + +## Background + +ORT has two `EpLibrary` implementations today: + +- `EpLibraryPlugin` (`onnxruntime/core/session/plugin_ep/ep_library_plugin.cc`) loads a shared library, resolves the + `CreateEpFactories` and `ReleaseEpFactory` symbols, and drives the factory lifecycle. +- `EpLibraryInternal` (`onnxruntime/core/session/plugin_ep/ep_library_internal.cc`) wraps an in-tree + `IExecutionProvider`, including `EpLibraryInternal::CreateWebGpuEp`. + +There is no `EpLibrary` that accepts factory entry points directly, so a provider compiled into the host cannot reach +the plugin path. + +This blocks ORT Web specifically. `cmake/onnxruntime_providers_webgpu.cmake` raises a `FATAL_ERROR` for the WebGPU +shared-module build under Emscripten, so static linking is the only way a Wasm build can ever reach the plugin +boundary. + +## Goals + +- Register a plugin EP that is linked into the host, reusing the existing plugin registration path after the point + where symbol lookup would occur. +- Present statically linked plugin code with the same runtime conditions as dynamically loaded plugin code. +- Require no change to provider sources between the two linkage modes. +- Avoid new host call sites for in-tree providers. +- Keep the shared-library path byte-for-byte unchanged at runtime. + +## Non-goals + +- A public C API for registering static plugin EPs. Deferred until a provider is genuinely out-of-tree. +- Removal of the EP API adapters. Tracked separately. +- Removal of the direct `IExecutionProvider` WebGPU path. That is the final step of the workstream, not this change. +- Support for static plugin EPs in minimal builds. `cmake/onnxruntime_session.cmake` excludes all of + `core/session/plugin_ep/` when `onnxruntime_MINIMAL_BUILD` is set. + +## Decisions + +### D1: Registration mechanism is a link-time registry in ORT core + +ORT core holds a hand-written, `#if`-guarded list of statically linked plugin EP entry points, mirroring the shape of +`EpLibraryInternal::CreateInternalEps`. A new `EpLibraryStaticPlugin` accepts the entry points directly and reuses +`EpLibraryPlugin`'s factory lifecycle logic without the dynamic-library load and unload steps. + +Rationale: every host — `onnxruntime_test_all`, the Python bindings, `onnxruntime.dll`, and Wasm — gets the provider +with no per-host call site. + +Alternatives considered: + +- CMake-generated registry using an X-macro. Rejected: added build-system machinery for generality that may never be + needed while providers remain in-tree. +- A public `RegisterStaticExecutionProviderLibrary` C API called by each host. Rejected for now: it requires a call + site in every statically linked host and introduces a window in which `GetEpDevices` reports no devices. Revisit + when an out-of-tree provider needs static linking. + +### D2: Entry point names are macro-parameterized + +`onnxruntime/core/providers/webgpu/ep/api.cc` is parameterized by `ORT_PLUGIN_EP_ENTRY_POINT_PREFIX`. + +The shared build must continue to export the unprefixed `CreateEpFactories` and `ReleaseEpFactory`, because +`EpLibraryPlugin::Load` resolves those exact names. The static build emits prefixed variants, for example +`WebGpu_CreateEpFactories`, so that multiple static plugins can coexist in one binary. + +### D3: No new EP library ABI hook for teardown + +All teardown remains in `ReleaseEpFactory`. Process-global shutdown that belongs to the host — currently +`google::protobuf::ShutdownProtobufLibrary()` — is compiled out under `ORT_PLUGIN_EP_OWNS_PROCESS_GLOBALS`. + +Rationale: a new optional entry point cannot be relied upon, because an older ORT would load a newer plugin and +silently ignore it. Behavior that varies with the host version is worse than a compile-time decision. + +The legacy static WebGPU cleanup block in `onnxruntime/core/session/ort_env.cc` is already guarded by +`defined(USE_WEBGPU) && !defined(ORT_USE_EP_API_ADAPTERS)`. The static plugin configuration defines +`ORT_USE_EP_API_ADAPTERS`, so the block compiles out on its own and needs no edit. It must be kept for the default +internal-EP WebGPU build. + +Consequence to test for: static linking has no library unload, so provider process-global state survives +unregistration. Registering, running, unregistering, re-registering, and running again is the primary regression case. + +### D4: One new durable build option + +- `onnxruntime_USE_EP_API_ADAPTERS` is unchanged. It is transitional and is retired when WebGPU moves fully onto the + public plugin API. +- `onnxruntime_WEBGPU_STATIC_PLUGIN` is new and durable. It selects linkage into the host rather than a loadable + module. +- `onnxruntime_WEBGPU_LINKED_INTO_HOST` is derived and used at the linkage sites. + +`onnxruntime_USE_EP_API_ADAPTERS` currently conflates three meanings: compiling against the adapters, building a +separate module, and registering by path in tests. The static plugin configuration answers the first yes and the +other two no, so only the linkage and test sites move to the derived option. The global `add_compile_definitions` in +`cmake/CMakeLists.txt` stays as-is, and no provider sources change. + +`cmake/onnxruntime_providers_webgpu.cmake` gains a third arm. The Emscripten and `onnxruntime_BUILD_CACHE` +`FATAL_ERROR`s narrow to the shared-module case, which is what unblocks ORT Web. + +`build.py` gains `--use_webgpu static_plugin`. + +The CUDA plugin is unaffected: it is gated by `onnxruntime_BUILD_CUDA_EP_AS_PLUGIN` and sets +`ORT_USE_EP_API_ADAPTERS` as a private target compile definition, never referencing the CMake option. CUDA's split +between role (`BUILD_CUDA_EP_AS_PLUGIN`) and mechanism (`ORT_USE_EP_API_ADAPTERS`) is the in-tree template for +WebGPU's eventual cleanup. + +### D5: Test infrastructure treats the library path as optional + +In `onnxruntime/test/unittest_util/test_dynamic_plugin_ep.cc`, `InitializationConfig::ep_library_path` becomes +optional. When absent, library registration is skipped because ORT core has already performed it, and the RAII +registration handle is left empty. The existing handle deleter already tolerates an empty handle. Device selection, +de-duplication, and factory creation are unchanged. + +Virtual devices are enabled through the `allow_virtual_devices` environment configuration entry +(`kOrtEnvAllowVirtualDevices`) supplied at environment creation, rather than the `.virtual` registration-name suffix. +The suffix is unavailable because ORT core chooses the registration name for statically linked providers. + +The `dynamic_plugin_ep_infra` naming becomes inaccurate. Renaming is deferred to a separate mechanical change. + +### D6: Static plugin EPs are registered after the environment is published + +This is the subtlest decision and the one that constrains the others. + +**Problem.** The natural insertion point is next to `CreateAndRegisterInternalEps` inside `Environment::Initialize`. +Registration enumerates devices, which calls provider code: + +``` +OrtEnv::GetOrCreateInstance() + lock(m_) non-recursive + Environment::Create() -> Initialize() + Environment::CreateAndRegisterStaticPluginEps() + Environment::RegisterExecutionProviderLibrary() + EpInfo::Create -> factory GetSupportedDevices() + webgpu::ep::Factory::GetSupportedDevices() provider code + Api().ep.GetEnvConfigEntries() + OrtEnv::TryGetInstance() + lock(m_) self-deadlock +``` + +`Environment::CreateAndRegisterInternalEps` already documents this hazard, and the internal WebGPU factory avoids it +by capturing `allow_virtual_devices` at construction. A plugin factory cannot use that workaround, because the public +API is all it has. + +The problem is not confined to one function. Plugin EPs receive the full `OrtApi`, so any API that resolves the +environment is affected, including `CreateEnv`, which re-enters `GetOrCreateInstance` and deadlocks on the same mutex. + +**Decision.** `OrtEnv::GetOrCreateInstance` publishes `p_instance_` and takes its own reference *before* invoking +`Environment::CreateAndRegisterStaticPluginEps`, and `OrtEnv::m_` becomes a `std::recursive_mutex`. Registration +therefore runs against a fully constructed and published environment. + +Properties: + +- The entire `OrtApi` behaves normally for provider code. `TryGetInstance` finds a published instance, and a + re-entrant `CreateEnv` acquires the recursive lock and increments the reference count. +- No new contract for provider authors, and nothing for a future API author to remember. +- No race. Other threads block on `m_` for the duration, which is already true across `Environment::Create`. +- The reference count cannot reach zero mid-construction, because the creating thread's reference is taken first. +- Provider sources need no conditional compilation for the two linkage modes. + +Costs and limits: + +- Failure during registration requires explicit teardown of the just-published instance. +- A recursive mutex is normally undesirable. Here it states the actual invariant: the environment creation path can + legitimately re-enter the environment accessor on the same thread. +- Callers of `onnxruntime::Environment::Create` do not get static plugin EPs. In the tree this is one production call + site plus tests and orttraining sample binaries. +- Provider code that starts a thread which calls `CreateEnv` and then joins it will deadlock. This is already true of + any code running inside `Environment::Create`. + +The call to `Environment::CreateAndRegisterStaticPluginEps` therefore lives in `ort_env.cc`. Static plugin +registration is a process-singleton concern tied to the `OrtEnv` lifetime, so the singleton wrapper is its correct +home. + +Alternatives considered: + +- Thread-local pointer to the environment under construction, consulted by `GetEnvConfigEntries`. Rejected: it + addresses one function, while the reachable surface is the whole `OrtApi`. +- Defer device enumeration until after environment creation. Rejected: lazy initialization moves registration + failures from environment creation to the first `GetEpDevices` call, which is a worse place to report them. +- Register outside the lock after publishing. Rejected: another thread can observe a published environment whose + static EPs are not yet registered, and avoiding that requires a second lock and a completion barrier. + +`onnxruntime::Environment` remains directly constructible, and tests rely on that. `OrtEnv::GetOrCreateInstance` +returns any existing instance and ignores the logging manager, threading options, and configuration entries passed by +later callers, so a test needing specific threading or logging configuration must bypass it. Those tests are +deliberately opting out of process-singleton semantics and should not receive static plugin EPs. + +### D7: The statically linked build does not use manual C++ API initialization + +`include/onnxruntime/ep/api.h` force-enables `ORT_API_MANUAL_INIT` around its include of `onnxruntime_cxx_api.h`, and +`onnxruntime::ep::ApiInit` calls `Ort::InitApi(ort_api)`. That is required for a plugin EP shared library, which must +not call `OrtGetApiBase()` itself. + +`onnxruntime_cxx_api.h` emits `#pragma detect_mismatch("ORT_API_MANUAL_INIT", ...)` on MSVC, so every translation +unit linked into one binary must agree. The statically linked plugin EP is linked with ORT core and with test code, +neither of which uses manual initialization, so forcing it on produces `LNK2038` for every EP object file. + +The build therefore defines `ORT_PLUGIN_EP_STATICALLY_LINKED` on the statically linked plugin EP target. Under that +macro `ep/api.h` includes `onnxruntime_cxx_api.h` unmodified and `ApiInit` skips `Ort::InitApi`. The C++ API then +default-initializes from `OrtGetApiBase()->GetApi(ORT_API_VERSION)`, which resolves in-process and yields the same +`OrtApi` that `ApiInit` would have installed, because the EP and ORT are the same binary and therefore the same +version. `onnxruntime::ep::ApiPtrs` is still populated from the `OrtApiBase*` that ORT passes to `CreateEpFactories`, +so the EP's own API access is unchanged. + +The macro is generic rather than WebGPU-specific because `ep/api.h` is shared with the CUDA plugin EP. + +## Implementation order + +1. `EpLibraryStaticPlugin`, factoring the shared factory lifecycle out of `EpLibraryPlugin`. +2. The static registry and `Environment::CreateAndRegisterStaticPluginEps`, with the `ort_env.cc` + publish-then-register change. +3. `ep/api.cc` entry point prefixing and host-owned process-global guards. +4. Build option and `build.py` changes. +5. Test infrastructure changes. +6. `ORT_PLUGIN_EP_STATICALLY_LINKED` in `include/onnxruntime/ep/api.h`, per D7. +7. Applying the WebGPU plugin EP test definitions to `onnxruntime_provider_test` as well as `onnxruntime_test_all`, + via a shared `onnxruntime_set_webgpu_plugin_ep_test_definitions` cmake function. `onnxruntime_provider_test` holds + the operator tests, so without this the plugin path has no operator test coverage. It also needs + `ORT_UNIT_TEST_HAS_WEBGPU_STATIC_PLUGIN_EP` in its own right, because that macro is what enables the + `allow_virtual_devices` environment configuration entry at environment creation in `test_main.cc`. + +## Follow-up cleanup + +`Environment::CreateAndRegisterInternalEps` stays where it is, at the end of `Environment::Initialize`. Internal EP +factories are ORT-core code that does not re-enter the `OrtEnv` singleton, so they are not subject to the constraint +in D6. Moving them would also remove EP devices from every directly constructed `Environment`, and `InferenceSession` +reads `Environment::GetOrtEpDevices` directly. For static plugin EPs the resulting gap is unimplemented new +functionality; for internal EPs it would be a silent regression. + +The `allow_virtual_devices` parameter threaded through `EpLibraryInternal::CreateInternalEps` exists only for the +internal WebGPU EP factory, which cannot query the environment at `GetSupportedDevices` time. The comment in +`CreateAndRegisterInternalEps` documents that constraint. Both can be deleted once WebGPU is no longer an internal EP: +the remaining internal EPs are CPU, kept deliberately as a special case, and DML, which is no longer maintained. +Neither takes the parameter. In a build that defines `ORT_USE_EP_API_ADAPTERS` this is already true, since the +internal WebGPU EP is compiled out and the parameter reaches `ORT_UNUSED_PARAMETER`. + +This does not affect the `.virtual` registration-name suffix or the `allow_virtual_devices` environment +configuration entry, which remain in use for dynamically registered plugin libraries. + +## Risks and open questions + +- Whether adapter-compiled WebGPU objects link into ORT core without duplicate-symbol or ODR problems. One real + instance was found and resolved: the `ORT_API_MANUAL_INIT` `detect_mismatch` guard, see D7. Otherwise resolved: + `onnxruntime_test_all` and `onnxruntime_provider_test` both link cleanly against the statically linked EP. +- Provider process-global state surviving unregistration, since static linking has no unload event. Covered by the + re-registration regression case in D3. Not yet exercised: `OrtEnv` is a reference-counted process singleton and the + unit test main holds a reference for the lifetime of the process, so a second `Environment` construction (and + therefore a second `CreateEpFactories` call) is not reachable from the existing test binaries. This needs either a + dedicated single-test process or a test that fully releases and recreates the environment. +- Whether any shipped ORT Web configuration uses an extended-minimal build, which excludes plugin EP infrastructure + entirely. +- Whether any `std::condition_variable` is paired with `OrtEnv::m_`, which would require `condition_variable_any` + under a recursive mutex. +- The existing rejection of `onnxruntime_BUILD_DAWN_SHARED_LIBRARY` together with the adapters. Initially left + rejecting both plugin kinds. +- Binary size and dead-code elimination for statically linked providers, per the workstream completion criteria. + +## Validation status + +Verified on Windows with `--use_webgpu static_plugin`, Debug: + +- `onnxruntime_test_all` and `onnxruntime_provider_test` build and link. +- The statically linked WebGPU plugin EP is registered by ORT core during `Ort::Env` creation, and its `OrtEpDevice` + is enumerated and selectable, confirming the D6 publish-then-register ordering resolves the self-deadlock. +- `InferenceSessionTests.WebGpuVirtualDeviceCompileOnlyEndToEnd` and + `InferenceSessionTests.WebGpuVirtualDeviceRejectedWithoutCompileOnly` pass, which covers session creation and EP + instantiation end to end through the public V2 API on the static plugin path. + +Not yet verified: operator execution. The development machine is a virtual machine with no real GPU, so only the +virtual WebGPU device is surfaced and non-compile-only sessions are rejected up front. This was confirmed to be a +property of the host rather than of static linking by building the shared library plugin configuration +(`--use_webgpu shared_lib`) on the same host and running the same tests: it passes and fails exactly the same set +(`WebGpuVirtualDeviceCompileOnlyEndToEnd` and `WebGpuVirtualDeviceRejectedWithoutCompileOnly` pass; +`WebGpuCompileOnlySkipsFinalization` and `TestStrictShapeInference` fail with the identical virtual-device rejection +message). Operator coverage on either plugin path requires a host with a GPU. + +The shared library plugin configuration was also re-verified after the D7 change, since `include/onnxruntime/ep/api.h` +is shared with the CUDA plugin EP. It builds, and `onnxruntime_providers_webgpu.dll` still exports exactly the +unprefixed `CreateEpFactories` and `ReleaseEpFactory` required by `EpLibraryPlugin::Load`, per D2. + +## Resolution of workstream open questions + +> Should static factories be registered before environment creation or through environment construction options? + +Neither. They are registered by ORT core during `OrtEnv` creation, after the environment is constructed and +published. See D6. diff --git a/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md b/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md index a704ea097a5c3..e98952daeb12b 100644 --- a/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md +++ b/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md @@ -55,6 +55,9 @@ The effort is divided into four workstreams: | `test-conformance` | [Test ownership and operator conformance](test_ownership_and_conformance/test_ownership_and_conformance_workstream.md) | Existing coverage is preserved, every test has an owner, and portable conformance coverage protects the external provider | | `node-migration` | [Node plugin migration](node_plugin_migration/node_plugin_migration_workstream.md) | Existing bundled Node WebGPU support is replaced by an explicitly consumable plugin without regressing current users | +The detailed static plugin EP registration design is in +[Static Plugin EP Registration](plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md). + The detailed reusable conformance-suite design is in [Execution Provider Operator Conformance Suite](test_ownership_and_conformance/ep_operator_conformance_design.md). diff --git a/include/onnxruntime/core/session/environment.h b/include/onnxruntime/core/session/environment.h index 338dbf85ef664..3a9f074cc895b 100644 --- a/include/onnxruntime/core/session/environment.h +++ b/include/onnxruntime/core/session/environment.h @@ -140,6 +140,20 @@ class Environment { Status RegisterExecutionProviderLibrary(const std::string& registration_name, const ORTCHAR_T* lib_path); Status UnregisterExecutionProviderLibrary(const std::string& registration_name); + /** + * Register the plugin execution providers that are statically linked into the ORT binary. + * + * This must be called after the OrtEnv singleton that owns this Environment has been published, because a + * statically linked plugin EP uses the public ORT API, and any OrtEnv API it calls must be able to find the + * instance. See docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/ + * static_plugin_ep_registration_design.md. + * + * Note: this does not take mutex_. It is safe because the caller holds the OrtEnv creation mutex, so no other + * thread can obtain a reference to this Environment yet. Taking mutex_ here would deadlock if a plugin EP called + * an Environment API from its OrtEpFactory::GetSupportedDevices implementation. + */ + Status CreateAndRegisterStaticPluginEps(); + // convert an OrtEpFactory* to EpFactoryInternal* if possible. EpFactoryInternal* GetEpFactoryInternal(OrtEpFactory* factory) const { // we're comparing pointers so the reinterpret_cast should be safe diff --git a/include/onnxruntime/ep/api.h b/include/onnxruntime/ep/api.h index 5a6da2250a9cd..2520028639a4e 100644 --- a/include/onnxruntime/ep/api.h +++ b/include/onnxruntime/ep/api.h @@ -10,11 +10,22 @@ #include #include +// A plugin EP built as its own shared library must not call OrtGetApiBase() itself. It receives the OrtApiBase* from +// ORT and initializes the C++ API manually, so ORT_API_MANUAL_INIT is forced on for onnxruntime_cxx_api.h. +// +// When the plugin EP is statically linked into the ORT binary (ORT_PLUGIN_EP_STATICALLY_LINKED), OrtGetApiBase() is +// available in-process and the rest of the binary is compiled without ORT_API_MANUAL_INIT. MSVC's detect_mismatch +// guard in onnxruntime_cxx_api.h requires every translation unit in a binary to agree on ORT_API_MANUAL_INIT, so in +// that case leave it alone and let the C++ API initialize itself. +#if !defined(ORT_PLUGIN_EP_STATICALLY_LINKED) #pragma push_macro("ORT_API_MANUAL_INIT") #undef ORT_API_MANUAL_INIT #define ORT_API_MANUAL_INIT #include "onnxruntime_cxx_api.h" #pragma pop_macro("ORT_API_MANUAL_INIT") +#else +#include "onnxruntime_cxx_api.h" +#endif namespace onnxruntime { namespace ep { @@ -141,8 +152,11 @@ inline void ApiInit(const OrtApiBase* ort_api_base, const char* min_ort_version throw std::runtime_error("Failed to initialize EP API: GetEpApi or GetModelEditorApi returned null."); } - // Manual init for the C++ API +#if !defined(ORT_PLUGIN_EP_STATICALLY_LINKED) + // Manual init for the C++ API. When statically linked, the C++ API initializes itself from OrtGetApiBase() with + // ORT_API_VERSION, which is the same API pointer because the EP and ORT are the same binary. Ort::InitApi(ort_api); +#endif // Initialize globals detail::g_api_ptrs.emplace(*ort_api, *ep_api, *model_editor_api); diff --git a/onnxruntime/core/providers/webgpu/ep/api.cc b/onnxruntime/core/providers/webgpu/ep/api.cc index 9ee9cbad513dd..b795245026019 100644 --- a/onnxruntime/core/providers/webgpu/ep/api.cc +++ b/onnxruntime/core/providers/webgpu/ep/api.cc @@ -1,9 +1,15 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +// See the comment on ORT_PLUGIN_EP_STATICALLY_LINKED in include/onnxruntime/ep/api.h for why manual init of the C++ +// API is only used when this EP is built as its own shared library. +#if !defined(ORT_PLUGIN_EP_STATICALLY_LINKED) #define ORT_API_MANUAL_INIT #include "onnxruntime_cxx_api.h" #undef ORT_API_MANUAL_INIT +#else +#include "onnxruntime_cxx_api.h" +#endif #include #include @@ -11,8 +17,22 @@ #include "core/providers/webgpu/ep/factory.h" -// To make symbols visible on macOS/iOS -#ifdef __APPLE__ +// When this EP is statically linked into the host binary instead of being built as a separate plugin library, the +// entry points below are given a prefix (e.g. `WebGpu_`) so that multiple statically linked plugin EPs can coexist +// in one binary. The build defines ORT_PLUGIN_EP_ENTRY_POINT_PREFIX in that case. ORT core declares the prefixed +// names in onnxruntime/core/session/plugin_ep/ep_static_plugins.cc. +// +// The shared library build must leave the entry points unprefixed, because they are resolved by exact name. +#if defined(ORT_PLUGIN_EP_ENTRY_POINT_PREFIX) +#define ORT_PLUGIN_EP_ENTRY_POINT_CONCAT_IMPL(prefix, name) prefix##name +#define ORT_PLUGIN_EP_ENTRY_POINT_CONCAT(prefix, name) ORT_PLUGIN_EP_ENTRY_POINT_CONCAT_IMPL(prefix, name) +#define ORT_PLUGIN_EP_ENTRY_POINT(name) ORT_PLUGIN_EP_ENTRY_POINT_CONCAT(ORT_PLUGIN_EP_ENTRY_POINT_PREFIX, name) +#else +#define ORT_PLUGIN_EP_ENTRY_POINT(name) name +#endif + +// To make symbols visible on macOS/iOS. Only needed when this EP is built as a separate shared library. +#if defined(__APPLE__) && !defined(ORT_PLUGIN_EP_ENTRY_POINT_PREFIX) #define EXPORT_SYMBOL __attribute__((visibility("default"))) #else #define EXPORT_SYMBOL @@ -25,19 +45,22 @@ void CleanupKernelRegistries(); } // namespace webgpu } // namespace onnxruntime +#if defined(ORT_PLUGIN_EP_OWNS_PROCESS_GLOBALS) namespace google { namespace protobuf { void ShutdownProtobufLibrary(); } // namespace protobuf } // namespace google +#endif extern "C" { // // Public symbols // -EXPORT_SYMBOL OrtStatus* CreateEpFactories(const char* /*registration_name*/, const OrtApiBase* ort_api_base, - const OrtLogger* default_logger, - OrtEpFactory** factories, size_t max_factories, size_t* num_factories) noexcept { +EXPORT_SYMBOL OrtStatus* ORT_PLUGIN_EP_ENTRY_POINT(CreateEpFactories)( + const char* /*registration_name*/, const OrtApiBase* ort_api_base, + const OrtLogger* default_logger, + OrtEpFactory** factories, size_t max_factories, size_t* num_factories) noexcept { { // Note: We can't use the EXCEPTION_TO_RETURNED_STATUS_BEGIN/EXCEPTION_TO_RETURNED_STATUS_END macros around the // call to `onnxruntime::ep::ApiInit()` because they depend on the API to create `OrtStatus`. We need to create an @@ -93,7 +116,7 @@ EXPORT_SYMBOL OrtStatus* CreateEpFactories(const char* /*registration_name*/, co EXCEPTION_TO_RETURNED_STATUS_END } -EXPORT_SYMBOL OrtStatus* ReleaseEpFactory(OrtEpFactory* factory) noexcept { +EXPORT_SYMBOL OrtStatus* ORT_PLUGIN_EP_ENTRY_POINT(ReleaseEpFactory)(OrtEpFactory* factory) noexcept { EXCEPTION_TO_RETURNED_STATUS_BEGIN // STEP.1 - Release the factory delete static_cast(factory); @@ -107,8 +130,12 @@ EXPORT_SYMBOL OrtStatus* ReleaseEpFactory(OrtEpFactory* factory) noexcept { // STEP.4 - Destroy the global default logger wrapper ::onnxruntime::ep::adapter::LoggingManager::DestroyDefaultLogger(); - // STEP.5 - Shutdown protobuf library +#if defined(ORT_PLUGIN_EP_OWNS_PROCESS_GLOBALS) + // STEP.5 - Shutdown protobuf library. + // Only do this when this EP owns the process-global state, i.e. when it is built as a separate library. + // When statically linked into the host binary, protobuf is shared with the host, which owns its lifetime. google::protobuf::ShutdownProtobufLibrary(); +#endif return nullptr; EXCEPTION_TO_RETURNED_STATUS_END diff --git a/onnxruntime/core/session/environment.cc b/onnxruntime/core/session/environment.cc index 0a32c142f2a83..639a9de70eacd 100644 --- a/onnxruntime/core/session/environment.cc +++ b/onnxruntime/core/session/environment.cc @@ -22,6 +22,7 @@ #include "core/session/plugin_ep/ep_library_internal.h" #include "core/session/plugin_ep/ep_library_plugin.h" #include "core/session/plugin_ep/ep_library_provider_bridge.h" +#include "core/session/plugin_ep/ep_static_plugins.h" #include "core/session/ort_apis.h" #include "core/session/utils.h" @@ -643,6 +644,17 @@ Status Environment::CreateAndRegisterInternalEps() { return Status::OK(); } +Status Environment::CreateAndRegisterStaticPluginEps() { + // Note: intentionally does not take mutex_. See the declaration in environment.h for why. + auto static_plugin_ep_libraries = CreateStaticPluginEpLibraries(); + for (auto& ep_library : static_plugin_ep_libraries) { + const std::string registration_name = ep_library->RegistrationName(); + ORT_RETURN_IF_ERROR(RegisterExecutionProviderLibrary(registration_name, std::move(ep_library))); + } + + return Status::OK(); +} + Status Environment::RegisterExecutionProviderLibrary(const std::string& registration_name, const ORTCHAR_T* lib_path) { std::lock_guard lock{mutex_}; diff --git a/onnxruntime/core/session/ort_env.cc b/onnxruntime/core/session/ort_env.cc index 68ecd7c2c804e..ca855a7652f54 100644 --- a/onnxruntime/core/session/ort_env.cc +++ b/onnxruntime/core/session/ort_env.cc @@ -4,6 +4,7 @@ // this file contains implementations of the C API #include +#include #include "ort_env.h" #include "core/session/ort_apis.h" @@ -29,7 +30,7 @@ void CleanupWebGpuContexts(); OrtEnv* OrtEnv::p_instance_; int OrtEnv::ref_count_ = 0; -std::mutex OrtEnv::m_; +std::recursive_mutex OrtEnv::m_; OrtEnv::OrtEnv(std::unique_ptr value1) : value_(std::move(value1)) { @@ -57,7 +58,7 @@ OrtEnvPtr OrtEnv::GetOrCreateInstance(const OrtEnv::LoggingManagerConstructionIn onnxruntime::common::Status& status, const OrtThreadingOptions* tp_options, const OrtKeyValuePairs* config_entries) { - std::lock_guard lock(m_); + std::lock_guard lock(m_); if (!p_instance_) { std::unique_ptr lmgr; std::string name = lm_info.logid; @@ -89,6 +90,27 @@ OrtEnvPtr OrtEnv::GetOrCreateInstance(const OrtEnv::LoggingManagerConstructionIn // Use 'new' to allocate OrtEnv, as it will be managed by p_instance_ // and deleted in ReleaseEnv or leaked if g_is_process_shutting_down is true. p_instance_ = new OrtEnv(std::move(env)); + + // Take this caller's reference before running any execution provider code below, so that if a provider + // acquires and releases an OrtEnvPtr the instance isn't destroyed out from under us. + ++ref_count_; + +#if !defined(ORT_MINIMAL_BUILD) + // Register statically linked plugin EPs *after* p_instance_ is published. They use the public ORT API, so any + // OrtEnv API they call while enumerating their devices must be able to find the instance. m_ is recursive so + // that such a call from this thread doesn't self-deadlock. See + // docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md + status = p_instance_->GetEnvironment().CreateAndRegisterStaticPluginEps(); + + if (!status.IsOK()) { + OrtEnv* failed_instance = std::exchange(p_instance_, nullptr); + ref_count_ = 0; + delete failed_instance; + return OrtEnvPtr(nullptr, OrtEnv::Release); + } +#endif // !defined(ORT_MINIMAL_BUILD) + + return OrtEnvPtr(p_instance_, OrtEnv::Release); } ++ref_count_; @@ -104,7 +126,7 @@ void OrtEnv::Release(OrtEnv* env_ptr) { OrtEnv* instance_to_delete = nullptr; { // Scope for the lock guard - std::lock_guard lock(m_); + std::lock_guard lock(m_); assert(p_instance_ == env_ptr); --ref_count_; @@ -135,7 +157,7 @@ void OrtEnv::Release(OrtEnv* env_ptr) { /*static*/ OrtEnvPtr OrtEnv::TryGetInstance() { - std::lock_guard lock(m_); + std::lock_guard lock(m_); if (p_instance_) { ++ref_count_; diff --git a/onnxruntime/core/session/ort_env.h b/onnxruntime/core/session/ort_env.h index 2f6c270e9c5e7..47613ec99ebf3 100644 --- a/onnxruntime/core/session/ort_env.h +++ b/onnxruntime/core/session/ort_env.h @@ -80,7 +80,9 @@ struct OrtEnv { // tracking active users. It is set to nullptr when the last reference is released // (and not shutting down). static OrtEnv* p_instance_; - static std::mutex m_; + // Recursive because statically linked plugin EPs are registered while this mutex is held, and a plugin EP may + // call an OrtEnv API (e.g. OrtEpApi::GetEnvConfigEntries) from OrtEpFactory::GetSupportedDevices. + static std::recursive_mutex m_; static int ref_count_; std::unique_ptr value_; diff --git a/onnxruntime/core/session/plugin_ep/ep_library_plugin.cc b/onnxruntime/core/session/plugin_ep/ep_library_plugin.cc index 5197035e3e3e4..3e4487fc5c5d1 100644 --- a/onnxruntime/core/session/plugin_ep/ep_library_plugin.cc +++ b/onnxruntime/core/session/plugin_ep/ep_library_plugin.cc @@ -6,6 +6,7 @@ #include "core/common/logging/logging.h" #include "core/framework/error_code_helper.h" #include "core/session/environment.h" +#include "core/session/plugin_ep/ep_library_plugin_utils.h" namespace onnxruntime { Status EpLibraryPlugin::Load() { @@ -27,19 +28,7 @@ Status EpLibraryPlugin::Load() { ORT_RETURN_IF_ERROR(Env::Default().GetSymbolFromLibrary(handle_, "ReleaseEpFactory", reinterpret_cast(&release_fn_))); - // allocate buffer for EP to add factories to. library can add up to 4 factories. - std::vector factories{4, nullptr}; - - size_t num_factories = 0; - ORT_RETURN_IF_ERROR(ToStatusAndRelease(create_fn_(registration_name_.c_str(), OrtGetApiBase(), - logging::LoggingManager::DefaultLogger().ToExternal(), - factories.data(), factories.size(), &num_factories))); - - for (size_t i = 0; i < num_factories; ++i) { - factories_.push_back(factories[i]); - } - - return Status::OK(); + return ep_library_plugin_utils::CreateFactories(create_fn_, registration_name_, factories_); }(); } } @@ -70,36 +59,7 @@ Status EpLibraryPlugin::Unload() { // Call ReleaseEpFactory for all factories and unload the library. // Current implementation assumes any error is permanent so does not leave pieces around to re-attempt Unload. if (handle_) { - if (!factories_.empty()) { - try { - for (size_t idx = 0, end = factories_.size(); idx < end; ++idx) { - auto* factory = factories_[idx]; - if (factory == nullptr) { - continue; - } - - auto status = ToStatusAndRelease(release_fn_(factory)); - if (!status.IsOK()) { - // log it and treat it as released - LOGS_DEFAULT(ERROR) << "ReleaseEpFactory failed for: " << library_path_ << " with error: " - << status.ErrorMessage(); - } - - factories_[idx] = nullptr; // clear the pointer in case there's a failure before all are released - } - - factories_.clear(); - } catch (const std::exception& ex) { - LOGS_DEFAULT(ERROR) << "Failed releasing EP factories from " << library_path_ << ": " << ex.what(); - } - } - - // TODO: Is there a better way? Is it worth worrying about? - if (!factories_.empty()) { - LOGS_DEFAULT(ERROR) << "Unloading " << library_path_ << ". " << factories_.size() - << " factories were not released due to errors. This may cause memory leaks. " - "Please check the error details in the log."; - } + ep_library_plugin_utils::ReleaseFactories(release_fn_, factories_, library_path_.string()); ORT_RETURN_IF_ERROR(Env::Default().UnloadDynamicLibrary(handle_)); } diff --git a/onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.cc b/onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.cc new file mode 100644 index 0000000000000..10f4c021fb528 --- /dev/null +++ b/onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.cc @@ -0,0 +1,67 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +#include "core/session/plugin_ep/ep_library_plugin_utils.h" + +#include "core/common/logging/logging.h" +#include "core/framework/error_code_helper.h" + +namespace onnxruntime { +namespace ep_library_plugin_utils { + +Status CreateFactories(CreateEpApiFactoriesFn create_fn, const std::string& registration_name, + std::vector& factories) { + // allocate buffer for EP to add factories to. library can add up to 4 factories. + std::vector new_factories{4, nullptr}; + + size_t num_factories = 0; + ORT_RETURN_IF_ERROR(ToStatusAndRelease(create_fn(registration_name.c_str(), OrtGetApiBase(), + logging::LoggingManager::DefaultLogger().ToExternal(), + new_factories.data(), new_factories.size(), &num_factories))); + + factories.reserve(factories.size() + num_factories); + for (size_t i = 0; i < num_factories; ++i) { + factories.push_back(new_factories[i]); + } + + return Status::OK(); +} + +void ReleaseFactories(ReleaseEpApiFactoryFn release_fn, std::vector& factories, + std::string_view library_description) { + if (factories.empty()) { + return; + } + + try { + for (size_t idx = 0, end = factories.size(); idx < end; ++idx) { + auto* factory = factories[idx]; + if (factory == nullptr) { + continue; + } + + auto status = ToStatusAndRelease(release_fn(factory)); + if (!status.IsOK()) { + // log it and treat it as released + LOGS_DEFAULT(ERROR) << "ReleaseEpFactory failed for: " << library_description << " with error: " + << status.ErrorMessage(); + } + + factories[idx] = nullptr; // clear the pointer in case there's a failure before all are released + } + + factories.clear(); + } catch (const std::exception& ex) { + LOGS_DEFAULT(ERROR) << "Failed releasing EP factories from " << library_description << ": " << ex.what(); + } + + // TODO: Is there a better way? Is it worth worrying about? + if (!factories.empty()) { + LOGS_DEFAULT(ERROR) << "Unloading " << library_description << ". " << factories.size() + << " factories were not released due to errors. This may cause memory leaks. " + "Please check the error details in the log."; + } +} + +} // namespace ep_library_plugin_utils +} // namespace onnxruntime diff --git a/onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.h b/onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.h new file mode 100644 index 0000000000000..4f7255b5f8550 --- /dev/null +++ b/onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.h @@ -0,0 +1,38 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +#pragma once + +#include +#include +#include + +#include "core/common/status.h" +#include "core/session/onnxruntime_c_api.h" + +namespace onnxruntime { +namespace ep_library_plugin_utils { + +// Shared implementation for EpLibrary types that obtain OrtEpFactory instances from a plugin EP's +// CreateEpFactories and ReleaseEpFactory entry points. The entry points are resolved at load time for a +// dynamically loaded library and at link time for a statically linked one, but the calling convention and +// lifetime rules are the same either way. + +/// +/// Calls the CreateEpFactories entry point and appends the factories it produces to `factories`. +/// `factories` is left unmodified if the entry point fails. +/// +Status CreateFactories(CreateEpApiFactoriesFn create_fn, const std::string& registration_name, + std::vector& factories); + +/// +/// Calls the ReleaseEpFactory entry point for every factory in `factories` and clears it. +/// Errors are logged rather than returned because the caller cannot meaningfully recover: the current +/// implementation assumes any failure is permanent and does not leave pieces around to re-attempt a release. +/// `library_description` identifies the library in log messages. +/// +void ReleaseFactories(ReleaseEpApiFactoryFn release_fn, std::vector& factories, + std::string_view library_description); + +} // namespace ep_library_plugin_utils +} // namespace onnxruntime diff --git a/onnxruntime/core/session/plugin_ep/ep_library_static_plugin.cc b/onnxruntime/core/session/plugin_ep/ep_library_static_plugin.cc new file mode 100644 index 0000000000000..95cae21e0d238 --- /dev/null +++ b/onnxruntime/core/session/plugin_ep/ep_library_static_plugin.cc @@ -0,0 +1,45 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +#include "core/session/plugin_ep/ep_library_static_plugin.h" + +#include "core/common/common.h" +#include "core/session/plugin_ep/ep_library_plugin_utils.h" + +namespace onnxruntime { +Status EpLibraryStaticPlugin::Load() { + auto status = Status::OK(); + + ORT_TRY { + std::lock_guard lock{mutex_}; + if (factories_.empty()) { + status = ep_library_plugin_utils::CreateFactories(create_fn_, registration_name_, factories_); + } + } + ORT_CATCH(const std::exception& ex) { + ORT_HANDLE_EXCEPTION([&]() { + status = ORT_MAKE_STATUS(ONNXRUNTIME, FAIL, + "Failed to create factories for statically linked execution " + "provider '", + registration_name_, "' with error: ", ex.what()); + }); + } + + // Release anything that was created before the failure so a partial load does not leave the library + // half-initialized. There is no library handle to release. + if (!status.IsOK()) { + std::lock_guard lock{mutex_}; + ep_library_plugin_utils::ReleaseFactories(release_fn_, factories_, registration_name_); + } + + return status; +} + +Status EpLibraryStaticPlugin::Unload() { + std::lock_guard lock{mutex_}; + + ep_library_plugin_utils::ReleaseFactories(release_fn_, factories_, registration_name_); + + return Status::OK(); +} +} // namespace onnxruntime diff --git a/onnxruntime/core/session/plugin_ep/ep_library_static_plugin.h b/onnxruntime/core/session/plugin_ep/ep_library_static_plugin.h new file mode 100644 index 0000000000000..617b11baeca22 --- /dev/null +++ b/onnxruntime/core/session/plugin_ep/ep_library_static_plugin.h @@ -0,0 +1,55 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +#pragma once + +#include +#include +#include + +#include "core/session/plugin_ep/ep_library.h" + +namespace onnxruntime { +/// +/// EpLibraryStaticPlugin supports a plugin execution provider that is statically linked into the ORT binary. +/// +/// The provider implements the same CreateEpFactories and ReleaseEpFactory entry points that a dynamically loaded +/// plugin library exports, and uses the same public EP API, but the entry points are resolved at link time instead +/// of being looked up by name. There is no library to load or unload, so the provider's process-global state +/// outlives unregistration. +/// +class EpLibraryStaticPlugin : public EpLibrary { + public: + EpLibraryStaticPlugin(std::string registration_name, + CreateEpApiFactoriesFn create_fn, + ReleaseEpApiFactoryFn release_fn) + : registration_name_{std::move(registration_name)}, + create_fn_{create_fn}, + release_fn_{release_fn} { + } + + const char* RegistrationName() const override { + return registration_name_.c_str(); + } + + // LibraryPath() intentionally uses the base class implementation, which returns nullptr. There is no library + // file backing a statically linked provider. + + Status Load() override; + + const std::vector& GetFactories() override { + return factories_; + } + + Status Unload() override; + + ORT_DISALLOW_COPY_AND_ASSIGNMENT(EpLibraryStaticPlugin); + + private: + std::mutex mutex_; + const std::string registration_name_; + const CreateEpApiFactoriesFn create_fn_; + const ReleaseEpApiFactoryFn release_fn_; + std::vector factories_{}; +}; +} // namespace onnxruntime diff --git a/onnxruntime/core/session/plugin_ep/ep_static_plugins.cc b/onnxruntime/core/session/plugin_ep/ep_static_plugins.cc new file mode 100644 index 0000000000000..d2120b4c9a5f1 --- /dev/null +++ b/onnxruntime/core/session/plugin_ep/ep_static_plugins.cc @@ -0,0 +1,38 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +#include "core/session/plugin_ep/ep_static_plugins.h" + +#include "core/graph/constants.h" +#include "core/session/plugin_ep/ep_library_static_plugin.h" + +// Entry points of the plugin execution providers that are statically linked into this build. +// +// A statically linked provider compiles its CreateEpFactories and ReleaseEpFactory entry points with a provider +// specific prefix (see ORT_PLUGIN_EP_ENTRY_POINT_PREFIX in the provider's api.cc) so that several providers can be +// linked into one binary without their entry points colliding. Declare them here rather than including a provider +// header, so that ORT core does not take a compile time dependency on provider internals. +#if defined(USE_WEBGPU) && defined(ORT_WEBGPU_STATIC_PLUGIN) +extern "C" { +OrtStatus* WebGpu_CreateEpFactories(const char* registration_name, const OrtApiBase* ort_api_base, + const OrtLogger* default_logger, OrtEpFactory** factories, + size_t max_factories, size_t* num_factories) noexcept; +OrtStatus* WebGpu_ReleaseEpFactory(OrtEpFactory* factory) noexcept; +} +#endif + +namespace onnxruntime { +std::vector> CreateStaticPluginEpLibraries() { + std::vector> ep_libraries; + +#if defined(USE_WEBGPU) && defined(ORT_WEBGPU_STATIC_PLUGIN) + // Register under the EP name, matching what an internal EP does. ORT core picks the registration name because + // there is no application call site to supply one. + ep_libraries.push_back(std::make_unique(kWebGpuExecutionProvider, + &WebGpu_CreateEpFactories, + &WebGpu_ReleaseEpFactory)); +#endif + + return ep_libraries; +} +} // namespace onnxruntime diff --git a/onnxruntime/core/session/plugin_ep/ep_static_plugins.h b/onnxruntime/core/session/plugin_ep/ep_static_plugins.h new file mode 100644 index 0000000000000..7107bedcb3861 --- /dev/null +++ b/onnxruntime/core/session/plugin_ep/ep_static_plugins.h @@ -0,0 +1,17 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +#pragma once + +#include +#include + +#include "core/session/plugin_ep/ep_library.h" + +namespace onnxruntime { +/// +/// Creates an EpLibrary for each plugin execution provider that is statically linked into this build. +/// The returned libraries have not been loaded yet. +/// +std::vector> CreateStaticPluginEpLibraries(); +} // namespace onnxruntime diff --git a/onnxruntime/test/framework/inference_session_test.cc b/onnxruntime/test/framework/inference_session_test.cc index 9efbf8c390d62..299ba144fa373 100644 --- a/onnxruntime/test/framework/inference_session_test.cc +++ b/onnxruntime/test/framework/inference_session_test.cc @@ -523,8 +523,10 @@ TEST(InferenceSessionTests, WebGpuCompileOnlyUsesNoOpAllocator) { // End-to-end via the public V2 API in the *plugin* WebGPU build: select the virtual WebGPU OrtEpDevice and run a // compile-only session. // -// Relies on test_main.cc registering the WebGPU plugin EP under a ".virtual" name, whose suffix auto-enables the env -// config "allow_virtual_devices" so the factory surfaces a virtual GPU OrtEpDevice. +// Relies on test_main.cc arranging for the factory to surface a virtual GPU OrtEpDevice: in the shared-library +// plugin build by registering the EP library under a ".virtual" name, whose suffix auto-enables the env config +// "allow_virtual_devices"; in the static plugin build by setting that env config entry directly at environment +// creation (ORT core, not the test, registers the statically linked EP, so there is no registration name to suffix). // // It exercises the accepted (device-free) path even on a host that has a real GPU: device-free is driven by // session.compile_only, not by which device is selected, so no Dawn device is created. @@ -543,9 +545,10 @@ TEST(InferenceSessionTests, WebGpuVirtualDeviceCompileOnlyEndToEnd) { break; } } - // A virtual device must be present in this build (test_main.cc's ".virtual" registration enables it). + // A virtual device must be present in this build (see the comment above this test for how test_main.cc enables it). ASSERT_FALSE(selected.empty()) - << "Expected a virtual WebGPU EP device from test_main.cc's .virtual registration, but none was surfaced."; + << "Expected a virtual WebGPU EP device from test_main.cc's virtual device configuration, " + "but none was surfaced."; Ort::SessionOptions session_options; // session-level compile_only (NOT an EP option) -> drives the device-free context and stop-before-finalize. @@ -562,7 +565,7 @@ TEST(InferenceSessionTests, WebGpuVirtualDeviceCompileOnlyEndToEnd) { // internal factory): selecting the virtual WebGPU device for a normal (non-compile-only) session must be rejected // up front by the *adapter* factory's CreateEp with ORT_INVALID_ARGUMENT, rather than proceeding into Dawn to fail // obscurely with no real GPU behind the virtual device. Exercises the adapter factory's copy of the enforcement -// through the public V2 API. Depends on the same test_main.cc ".virtual" registration as +// through the public V2 API. Depends on the same test_main.cc virtual device configuration as // WebGpuVirtualDeviceCompileOnlyEndToEnd above (that's what surfaces the virtual device to select). TEST(InferenceSessionTests, WebGpuVirtualDeviceRejectedWithoutCompileOnly) { std::vector selected; @@ -577,10 +580,11 @@ TEST(InferenceSessionTests, WebGpuVirtualDeviceRejectedWithoutCompileOnly) { break; } } - // See WebGpuVirtualDeviceCompileOnlyEndToEnd: a virtual device must be present from test_main.cc's .virtual - // registration in this build, so fail (not skip) if none was surfaced. + // See WebGpuVirtualDeviceCompileOnlyEndToEnd: a virtual device must be present in this build, so fail (not skip) + // if none was surfaced. ASSERT_FALSE(selected.empty()) - << "Expected a virtual WebGPU EP device from test_main.cc's .virtual registration, but none was surfaced."; + << "Expected a virtual WebGPU EP device from test_main.cc's virtual device configuration, " + "but none was surfaced."; // Deliberately NOT setting session.compile_only -> a runnable session on a virtual device, which must be rejected. Ort::SessionOptions session_options; diff --git a/onnxruntime/test/unittest_main/test_main.cc b/onnxruntime/test/unittest_main/test_main.cc index 2693f0a93f050..4940d15df4f8b 100644 --- a/onnxruntime/test/unittest_main/test_main.cc +++ b/onnxruntime/test/unittest_main/test_main.cc @@ -42,6 +42,11 @@ #include "test/util/include/skipping_test_listener.h" #endif // defined(TEST_MAIN_ENABLE_DYNAMIC_PLUGIN_EP_USAGE) +#if defined(TEST_MAIN_ENABLE_DYNAMIC_PLUGIN_EP_USAGE) && defined(ORT_UNIT_TEST_HAS_WEBGPU_STATIC_PLUGIN_EP) +#define TEST_MAIN_ENABLE_WEBGPU_STATIC_PLUGIN_EP_USAGE +#include "core/session/onnxruntime_env_config_keys.h" +#endif + std::unique_ptr ort_env; // define environment variable name constants here @@ -82,7 +87,27 @@ extern "C" void ortenv_setup() { log_level = static_cast(*log_level_override); } +#if defined(TEST_MAIN_ENABLE_WEBGPU_STATIC_PLUGIN_EP_USAGE) + // The WebGPU plugin EP is statically linked into the ORT binary and is registered by ORT core while the + // environment is created, so "allow_virtual_devices" must be set here rather than being derived from a + // ".virtual" EP library registration name (there is no library registration call to name). This lets the + // WebGPU factory surface a virtual GPU device, which gives dynamic_plugin_ep_infra::Initialize() below a + // selectable WebGPU device so the binary doesn't exit at startup when GetEpDevices() would otherwise be + // empty (no real GPU). It does not make non-compile-only WebGPU tests runnable without a device. + Ort::KeyValuePairs env_config_entries{}; + env_config_entries.Add(kOrtEnvAllowVirtualDevices, "1"); + + OrtEnvCreationOptions env_creation_options{}; + env_creation_options.version = ORT_API_VERSION; + env_creation_options.logging_severity_level = static_cast(log_level); + env_creation_options.log_id = "Default"; + env_creation_options.threading_options = &tpo; + env_creation_options.config_entries = env_config_entries.GetConst(); + + ort_env.reset(new Ort::Env(&env_creation_options)); +#else ort_env.reset(new Ort::Env(&tpo, log_level, "Default")); +#endif #if defined(TEST_MAIN_ENABLE_DYNAMIC_PLUGIN_EP_USAGE) { @@ -142,6 +167,22 @@ extern "C" void ortenv_setup() { } #endif +#if defined(TEST_MAIN_ENABLE_WEBGPU_STATIC_PLUGIN_EP_USAGE) + if (!dynamic_plugin_ep_config_json.has_value()) { + // No "ep_library_path": ORT core already registered the statically linked WebGPU plugin EP when the + // environment above was created. + dynamic_plugin_ep_config_json.emplace( + "{\n" + " \"ep_library_registration_name\": \"WebGpuExecutionProvider\",\n" + // Enable int64 so the generic (EP-agnostic) int64 operator tests exercise the WebGPU + // int64 kernels on the plugin path too (mirrors DefaultWebGpuExecutionProvider on the + // internal EP path). The key is the un-prefixed provider option name for plugin EPs. + " \"default_ep_options\": { \"enableInt64\": \"1\" },\n" + " \"selected_ep_name\": \"WebGpuExecutionProvider\"\n" + "}"); + } +#endif + if (dynamic_plugin_ep_config_json.has_value()) { std::cout << "Initializing dynamic plugin EP infrastructure with configuration:\n" << *dynamic_plugin_ep_config_json << "\n"; diff --git a/onnxruntime/test/unittest_util/test_dynamic_plugin_ep.cc b/onnxruntime/test/unittest_util/test_dynamic_plugin_ep.cc index 3e4913e44fb86..4fb57635d22b0 100644 --- a/onnxruntime/test/unittest_util/test_dynamic_plugin_ep.cc +++ b/onnxruntime/test/unittest_util/test_dynamic_plugin_ep.cc @@ -84,9 +84,11 @@ Status ParseInitializationConfig(std::string_view json_str, InitializationConfig // required keys parsed_json.at("ep_library_registration_name").get_to(config.ep_library_registration_name); - parsed_json.at("ep_library_path").get_to(config.ep_library_path); // optional keys + // `ep_library_path` is not needed when the EP library is already registered, e.g. a plugin EP that is + // statically linked into the ORT binary and registered by ORT core at environment creation. + config.ep_library_path = parsed_json.value("ep_library_path", {}); config.default_ep_options = parsed_json.value("default_ep_options", {}); config.selected_ep_name = parsed_json.value("selected_ep_name", {}); config.selected_ep_device_indices = @@ -116,8 +118,13 @@ Status ParseInitializationConfig(std::string_view json_str, InitializationConfig Status Initialize(Ort::Env& env, InitializationConfig config) { ORT_RETURN_IF(IsInitialized(), "Already initialized."); - auto ep_library_registration_handle = RegisterPluginEpLibrary(env, config.ep_library_registration_name, - ToPathString(config.ep_library_path)); + // An empty `ep_library_path` means the EP library is already registered and this infrastructure should not + // register or unregister it. That is the case for a plugin EP that is statically linked into the ORT binary. + auto ep_library_registration_handle = + config.ep_library_path.empty() + ? PluginEpLibraryRegistrationHandle{nullptr, [](void*) {}} + : RegisterPluginEpLibrary(env, config.ep_library_registration_name, + ToPathString(config.ep_library_path)); ORT_RETURN_IF(config.selected_ep_device_indices.empty() == config.selected_ep_name.empty(), "Exactly one of selected_ep_device_indices or selected_ep_name should be specified."); diff --git a/onnxruntime/test/unittest_util/test_dynamic_plugin_ep.h b/onnxruntime/test/unittest_util/test_dynamic_plugin_ep.h index c3543962b80ef..6ece30801b316 100644 --- a/onnxruntime/test/unittest_util/test_dynamic_plugin_ep.h +++ b/onnxruntime/test/unittest_util/test_dynamic_plugin_ep.h @@ -40,6 +40,8 @@ inline constexpr std::string_view kWebGpuExecutionProviderPluginName{"WebGpuExec // Configuration for initializing the dynamic plugin EP infrastructure. struct InitializationConfig { std::string ep_library_registration_name{}; + // Path to the EP library to register. May be empty if the EP does not need to be registered by the test + // infrastructure, e.g. a plugin EP that is statically linked into the ORT binary and registered by ORT core. std::string ep_library_path{}; // Note: Exactly one of `selected_ep_name` or `selected_ep_device_indices` should be set. @@ -59,6 +61,8 @@ struct InitializationConfig { // Parses `InitializationConfig` from JSON. // The configuration JSON object should have keys and values that match the `InitializationConfig` fields. +// `ep_library_path` may be omitted for a plugin EP that is already registered (e.g. statically linked into the ORT +// binary and registered by ORT core). // E.g.: // { // "ep_library_registration_name": "example_plugin_ep", diff --git a/tools/ci_build/build.py b/tools/ci_build/build.py index ac7ee66feb3a2..e837a23e7841d 100644 --- a/tools/ci_build/build.py +++ b/tools/ci_build/build.py @@ -909,6 +909,9 @@ def generate_build_tree( cmake_args += ["-Donnxruntime_USE_EP_API_ADAPTERS=ON"] if args.build_wasm: raise BuildError("Only static library build of WebGPU EP is supported for WebAssembly build.") + elif args.use_webgpu == "static_plugin": + # Plugin EP that is linked into the ORT binary. + cmake_args += ["-Donnxruntime_USE_EP_API_ADAPTERS=ON", "-Donnxruntime_WEBGPU_STATIC_PLUGIN=ON"] if args.use_snpe: cmake_args += ["-Donnxruntime_USE_SNPE=ON"] diff --git a/tools/ci_build/build_args.py b/tools/ci_build/build_args.py index 736f697b226b3..829bf84c22004 100644 --- a/tools/ci_build/build_args.py +++ b/tools/ci_build/build_args.py @@ -99,7 +99,7 @@ def invalid_hetero_build() -> None: def _webgpu_verify_library_kind(library_kind: str) -> str: """Verifies the library kind for the WebGPU Execution Provider.""" - choices = ["shared_lib", "static_lib"] + choices = ["shared_lib", "static_lib", "static_plugin"] if library_kind not in choices: print("\nYou have specified an invalid library kind for WebGPU EP.") print(f"The invalid library kind was: {library_kind}") @@ -823,7 +823,10 @@ def add_execution_provider_args(parser: argparse.ArgumentParser) -> None: nargs="?", const="static_lib", type=_webgpu_verify_library_kind, - help="Enable WebGPU EP. Optionally specify 'static_lib' (default) or 'shared_lib'.", + help="Enable WebGPU EP. Optionally specify 'static_lib' (default), 'shared_lib', or 'static_plugin'. " + "'static_lib' builds the EP as an internal ORT EP. 'shared_lib' builds it as a separate plugin EP " + "library that is loaded at runtime. 'static_plugin' builds it as a plugin EP that is linked into the " + "ORT binary and registered at environment creation.", ) webgpu_group.add_argument( "--use_external_dawn", action="store_true", help="Use external Dawn dependency for WebGPU." From 32775c305a5d9c54d1f87f556942601b2442d313 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Tue, 25 Aug 2026 17:13:05 -0700 Subject: [PATCH 010/107] Add CI jobs for the WebGPU static plugin EP build The static_plugin configuration had no CI coverage, since both new cmake options default to OFF and every existing WebGPU job builds either the internal EP or the shared library plugin EP. Windows runs on the GPU pool and builds and tests, which is the only way to exercise the operator tests on this configuration. Linux is build-only, like the WebGPU job next to it, and mainly serves to compile the configuration with GCC as well as MSVC. It keeps --build_wheel because linking the EP into pybind11_state is specific to this configuration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/linux_webgpu.yml | 22 ++++++++ .github/workflows/windows_webgpu.yml | 78 ++++++++++++++++++++++++++++ 2 files changed, 100 insertions(+) diff --git a/.github/workflows/linux_webgpu.yml b/.github/workflows/linux_webgpu.yml index 234adcaee320c..44aad2400af75 100644 --- a/.github/workflows/linux_webgpu.yml +++ b/.github/workflows/linux_webgpu.yml @@ -37,6 +37,28 @@ jobs: secrets: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} # Pass token for reusable workflow needs (e.g., docker build action) + # Builds the same WebGPU plugin EP sources linked into the ORT binary rather than as a separate + # shared library. Build-only, like the job above; this exists mainly to compile the static plugin + # configuration with GCC as well as MSVC. See docs/design/webgpu_ep_extraction/ + # plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md + build-linux-webgpu-static-plugin-x64-release: + name: Build Linux WebGPU Static Plugin x64 Release + uses: ./.github/workflows/reusable_linux_build.yml + with: + pool_name: "onnxruntime-github-Ubuntu2204-AMD-CPU" # Build pool + build_config: Release + architecture: x64 + dockerfile_path: tools/ci_build/github/linux/docker/Dockerfile.manylinux2_28_webgpu + docker_image_repo: onnxruntimecpubuildpythonx64 + extra_build_flags: '--use_binskim_compliant_compile_flags --build_wheel --use_webgpu static_plugin --cmake_extra_defines onnxruntime_BUILD_UNIT_TESTS=ON' + python_path_prefix: 'PATH=/opt/python/cp310-cp310/bin:$PATH' + run_tests: false + upload_build_output: false + execution_providers: 'webgpu' + job_identifier: build-linux-webgpu-static-plugin-x64-release + secrets: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} # Pass token for reusable workflow needs (e.g., docker build action) + # TODO: the following test step is currently failing. Need to fix and re-enable it # test-linux-webgpu-x64-release: diff --git a/.github/workflows/windows_webgpu.yml b/.github/workflows/windows_webgpu.yml index 98357c2e8423f..4ef3ed6a226ea 100644 --- a/.github/workflows/windows_webgpu.yml +++ b/.github/workflows/windows_webgpu.yml @@ -228,6 +228,84 @@ jobs: ${{ github.workspace }}/RelWithDebInfo/RelWithDebInfo/dxcompiler.dll ${{ github.workspace }}/RelWithDebInfo/RelWithDebInfo/dxil.dll + # Same plugin EP sources as webgpu_plugin_build_x64_RelWithDebInfo, but linked into the ORT binary + # instead of built as a separate shared library. Registered by ORT core at environment creation. + # See docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/ + # static_plugin_ep_registration_design.md + webgpu_static_plugin_build_x64_RelWithDebInfo: + runs-on: [ + "self-hosted", + "1ES.Pool=onnxruntime-github-Win2022-GPU-A10", + "JobId=webgpu_static_plugin_build_x64_RelWithDebInfo-${{ github.run_id }}-${{ github.run_number }}-${{ github.run_attempt }}" + ] + timeout-minutes: 300 + env: + OnnxRuntimeBuildDirectory: ${{ github.workspace }} + setVcvars: true + ALLOW_RELEASED_ONNX_OPSET_ONLY: "0" + DocUpdateNeeded: false + NVIDIA_TF32_OVERRIDE: "0" + ONNXRUNTIME_TEST_GPU_DEVICE_ID: "0" + steps: + - name: Checkout + uses: actions/checkout@v6 + with: + fetch-depth: 0 + submodules: none + + - name: Setup Python 3.12 + uses: actions/setup-python@v6 + with: + python-version: "3.12" + architecture: x64 + + - name: Locate vcvarsall and Setup Env + uses: ./.github/actions/locate-vcvarsall-and-setup-env + with: + architecture: x64 + + - name: Install Visual Studio Spectre-mitigated libs + uses: ./.github/actions/install-vs-spectre-libs + with: + architecture: x64 + + - name: Install python modules + run: python -m pip install -r tools\ci_build\github\windows\python\requirements.txt + shell: cmd + working-directory: ${{ github.workspace }} + + - name: Setup Node.js + uses: actions/setup-node@v6 + with: + node-version: "20.x" + + - uses: actions/cache@v5 + id: onnx-node-tests-cache + with: + path: ${{ github.workspace }}/js/test/ + key: onnxnodetests-${{ hashFiles('js/scripts/prepare-onnx-node-tests.ts') }} + + # onnxruntime_BUILD_DAWN_SHARED_LIBRARY is deliberately not enabled here. It is incompatible with + # onnxruntime_USE_EP_API_ADAPTERS, which --use_webgpu static_plugin turns on. + - name: Build and Test + shell: pwsh + run: | + python.exe ${{ github.workspace }}\tools\ci_build\build.py ` + --config RelWithDebInfo ` + --build_dir ${{ github.workspace }} ` + --skip_submodule_sync ` + --parallel ` + --use_binskim_compliant_compile_flags ` + --cmake_generator "Visual Studio 17 2022" ` + --enable_onnx_tests ` + --use_webgpu static_plugin ` + --use_vcpkg --use_vcpkg_ms_internal_asset_cache ` + --cmake_extra_defines onnxruntime_BUILD_UNIT_TESTS=ON onnxruntime_ENABLE_DAWN_BACKEND_D3D12=1 onnxruntime_ENABLE_DAWN_BACKEND_VULKAN=1 + + if ($lastExitCode -ne 0) { + exit $lastExitCode + } + webgpu_external_dawn_build_x64_RelWithDebInfo: runs-on: [ "self-hosted", From 49855f6ef7124874a2e8574cb98ee25270a6bf31 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Wed, 26 Aug 2026 10:53:03 -0700 Subject: [PATCH 011/107] Fix GCC maybe-uninitialized errors in the EP adapter attribute accessors The EP adapter OpKernelInfo defines GetAttr()/GetAttrOrDefault() inline in the header, so GCC can see their failure and exception paths and reports -Werror=maybe-uninitialized for values that are only read when the lookup succeeded. The in-tree OpKernelInfo hides these paths behind an out-of-line definition, which is why this only shows up when building the EP adapters with GCC. Value-initialize the temporaries in the accessors and in the WebGPU kernel constructors that pass a local by pointer and read it back. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- include/onnxruntime/ep/adapter/op_kernel_info.h | 6 ++++-- onnxruntime/core/providers/webgpu/math/gemm.h | 4 ++-- onnxruntime/core/providers/webgpu/math/softmax.h | 2 +- onnxruntime/core/providers/webgpu/math/top_k.h | 2 +- onnxruntime/core/providers/webgpu/tensor/cast.h | 2 +- 5 files changed, 9 insertions(+), 7 deletions(-) diff --git a/include/onnxruntime/ep/adapter/op_kernel_info.h b/include/onnxruntime/ep/adapter/op_kernel_info.h index 417ebd4adf7a2..2e03caa60cdc9 100644 --- a/include/onnxruntime/ep/adapter/op_kernel_info.h +++ b/include/onnxruntime/ep/adapter/op_kernel_info.h @@ -154,7 +154,9 @@ struct OpKernelInfo { template [[nodiscard]] T GetAttrOrDefault(const std::string& name, const T& default_value) const { - T tmp; + // Value-initialize. These accessors are fully inline here, so the compiler can see the failure and + // exception paths of GetAttr() and warns about a possibly uninitialized read without it. + T tmp{}; return GetAttr(name, &tmp).IsOK() ? tmp : default_value; } template @@ -164,7 +166,7 @@ struct OpKernelInfo { } template [[nodiscard]] T GetAttr(const std::string& name) const { - T value; + T value{}; ORT_THROW_IF_ERROR(GetAttr(name, &value)); return value; } diff --git a/onnxruntime/core/providers/webgpu/math/gemm.h b/onnxruntime/core/providers/webgpu/math/gemm.h index bf97ff0f2b994..fcffc523cfd58 100644 --- a/onnxruntime/core/providers/webgpu/math/gemm.h +++ b/onnxruntime/core/providers/webgpu/math/gemm.h @@ -60,11 +60,11 @@ class Gemm final : public WebGpuKernel { }; Gemm(const OpKernelInfo& info) : WebGpuKernel(info) { - int64_t transA_temp; + int64_t transA_temp = 0; info.GetAttrOrDefault("transA", &transA_temp, static_cast(0)); transA_ = transA_temp != 0; - int64_t transB_temp; + int64_t transB_temp = 0; info.GetAttrOrDefault("transB", &transB_temp, static_cast(0)); transB_ = transB_temp != 0; diff --git a/onnxruntime/core/providers/webgpu/math/softmax.h b/onnxruntime/core/providers/webgpu/math/softmax.h index 532a56ff0be41..cae5581e9ff2f 100644 --- a/onnxruntime/core/providers/webgpu/math/softmax.h +++ b/onnxruntime/core/providers/webgpu/math/softmax.h @@ -15,7 +15,7 @@ class Softmax final : public WebGpuKernel { public: Softmax(const OpKernelInfo& info) : WebGpuKernel{info} { opset_ = info.node().SinceVersion(); - int64_t axis; + int64_t axis = 0; Status status = info.GetAttr("axis", &axis); if (status.IsOK()) { diff --git a/onnxruntime/core/providers/webgpu/math/top_k.h b/onnxruntime/core/providers/webgpu/math/top_k.h index 6652357ec71cc..08b5a2372c6ae 100644 --- a/onnxruntime/core/providers/webgpu/math/top_k.h +++ b/onnxruntime/core/providers/webgpu/math/top_k.h @@ -82,7 +82,7 @@ class TopK final : public WebGpuKernel { info.GetAttrOrDefault("largest", &largest_, 1); info.GetAttrOrDefault("sorted", &sorted_, 1); if (opset_ <= 9) { - int64_t k_temp; + int64_t k_temp = 0; ORT_ENFORCE(info.GetAttr("k", &k_temp).IsOK()); attr_k_ = k_temp; } diff --git a/onnxruntime/core/providers/webgpu/tensor/cast.h b/onnxruntime/core/providers/webgpu/tensor/cast.h index f86771854c2b6..6ff0e7cd40590 100644 --- a/onnxruntime/core/providers/webgpu/tensor/cast.h +++ b/onnxruntime/core/providers/webgpu/tensor/cast.h @@ -36,7 +36,7 @@ class CastProgram final : public Program { class Cast final : public WebGpuKernel { public: Cast(const OpKernelInfo& info) : WebGpuKernel(info) { - int64_t to; + int64_t to = 0; Status status = info.GetAttr("to", &to); ORT_ENFORCE(status.IsOK(), "Attribute to is not set."); to_ = onnxruntime::narrow(to); From d47f00a0a66d86052311f57f13ce71aee451d0dd Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Wed, 26 Aug 2026 10:53:13 -0700 Subject: [PATCH 012/107] Update dynamic plugin EP test config parsing tests `ep_library_path` is now optional in the dynamic plugin EP test infrastructure config, so that a statically linked plugin EP that ORT core has already registered can be selected. Update the test that asserted it was required, and add coverage for omitting it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../test/framework/dynamic_plugin_ep_test.cc | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/onnxruntime/test/framework/dynamic_plugin_ep_test.cc b/onnxruntime/test/framework/dynamic_plugin_ep_test.cc index 2453096fa8094..705233fe22a22 100644 --- a/onnxruntime/test/framework/dynamic_plugin_ep_test.cc +++ b/onnxruntime/test/framework/dynamic_plugin_ep_test.cc @@ -72,10 +72,27 @@ TEST(DynamicPluginEpInfraTest, ParseInitializationConfigDefaultsUnsetOptionalFie EXPECT_TRUE(config.tests_to_skip.empty()); } +TEST(DynamicPluginEpInfraTest, ParseInitializationConfigAllowsMissingEpLibraryPath) { + // `ep_library_path` may be omitted when the EP library is already registered, e.g. a statically linked plugin EP. + constexpr std::string_view kConfigJson = R"json( +{ + "ep_library_registration_name": "WebGpuExecutionProvider", + "selected_ep_name": "WebGpuExecutionProvider" +} +)json"; + + dynamic_plugin_ep_test_infra::InitializationConfig config{}; + ASSERT_STATUS_OK(dynamic_plugin_ep_test_infra::ParseInitializationConfig(kConfigJson, config)); + + EXPECT_EQ(config.ep_library_registration_name, "WebGpuExecutionProvider"); + EXPECT_TRUE(config.ep_library_path.empty()); + EXPECT_EQ(config.selected_ep_name, "WebGpuExecutionProvider"); +} + TEST(DynamicPluginEpInfraTest, ParseInitializationConfigRejectsMissingRequiredFields) { constexpr std::string_view kConfigJson = R"json( { - "ep_library_registration_name": "CUDAExecutionProvider" + "ep_library_path": "/tmp/libonnxruntime_providers_cuda.so" } )json"; From 1980e901ec74dab0753f8d4a17720c9bb7cc293d Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Wed, 26 Aug 2026 10:54:17 -0700 Subject: [PATCH 013/107] Record CI coverage in the static plugin EP registration design doc Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../static_plugin_ep_registration_design.md | 20 +++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md index b2e42d668aedc..71daa7d207048 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -273,8 +273,8 @@ Verified on Windows with `--use_webgpu static_plugin`, Debug: `InferenceSessionTests.WebGpuVirtualDeviceRejectedWithoutCompileOnly` pass, which covers session creation and EP instantiation end to end through the public V2 API on the static plugin path. -Not yet verified: operator execution. The development machine is a virtual machine with no real GPU, so only the -virtual WebGPU device is surfaced and non-compile-only sessions are rejected up front. This was confirmed to be a +Not yet verified locally: operator execution. The development machine is a virtual machine with no real GPU, so only +the virtual WebGPU device is surfaced and non-compile-only sessions are rejected up front. This was confirmed to be a property of the host rather than of static linking by building the shared library plugin configuration (`--use_webgpu shared_lib`) on the same host and running the same tests: it passes and fails exactly the same set (`WebGpuVirtualDeviceCompileOnlyEndToEnd` and `WebGpuVirtualDeviceRejectedWithoutCompileOnly` pass; @@ -285,6 +285,22 @@ The shared library plugin configuration was also re-verified after the D7 change is shared with the CUDA plugin EP. It builds, and `onnxruntime_providers_webgpu.dll` still exports exactly the unprefixed `CreateEpFactories` and `ReleaseEpFactory` required by `EpLibraryPlugin::Load`, per D2. +### CI coverage + +No pre-existing CI leg built this configuration, since both new CMake options default to `OFF`. Two jobs were added: + +- `webgpu_static_plugin_build_x64_RelWithDebInfo` in `.github/workflows/windows_webgpu.yml`, which builds and runs the + tests on a GPU-equipped pool. This closes the operator execution gap above. Note that it must not set + `onnxruntime_BUILD_DAWN_SHARED_LIBRARY=ON`, which is incompatible with `onnxruntime_USE_EP_API_ADAPTERS`. +- `build-linux-webgpu-static-plugin-x64-release` in `.github/workflows/linux_webgpu.yml`, build only, mirroring its + sibling job. This is the only GCC coverage of the EP adapters, and the only coverage of the `onnxruntime_python.cmake` + change, hence `--build_wheel` is kept. + +The first run of the Linux job surfaced `-Werror=maybe-uninitialized` errors. The EP adapter `OpKernelInfo` defines its +attribute accessors inline in the header, so GCC can see their failure and exception paths, whereas the in-tree +`OpKernelInfo` hides them behind an out-of-line definition. Any value that is only read when the attribute lookup +succeeded must be value-initialized. + ## Resolution of workstream open questions > Should static factories be registered before environment creation or through environment construction options? From 994e00bb449636e15c7112610c1f2e18e913e526 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Wed, 26 Aug 2026 13:20:12 -0700 Subject: [PATCH 014/107] Suppress GCC maybe-uninitialized on the WebGPU EP adapter build The EP adapter types are defined entirely inline in headers, so GCC sees through the failure and exception paths of accessors like OpKernelInfo::GetAttr<> and falsely warns about values that are only read once the call succeeded. It also reports false positives deep inside absl::InlinedVector for TensorShapeVector locals that are always constructed, which cannot be addressed at the source. Apply the same target scoped suppression the CUDA plugin EP already uses. It is limited to GCC and to the EP adapter build so that other builds stay strict. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- cmake/onnxruntime_providers_webgpu.cmake | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/cmake/onnxruntime_providers_webgpu.cmake b/cmake/onnxruntime_providers_webgpu.cmake index 5b6e45e8f57e0..a2fd7d5b8dd0b 100644 --- a/cmake/onnxruntime_providers_webgpu.cmake +++ b/cmake/onnxruntime_providers_webgpu.cmake @@ -166,6 +166,15 @@ target_precompile_headers(onnxruntime_providers_webgpu PRIVATE "${REPO_ROOT}/include/onnxruntime/ep/adapters.h" ) + + # Suppress -Werror=maybe-uninitialized. The EP adapter types are defined entirely inline in headers, + # so GCC sees through the failure and exception paths of accessors like OpKernelInfo::GetAttr<> and + # falsely warns about values that are only read once the call has succeeded. It also produces false + # positives deep inside absl::InlinedVector for locals that are always constructed. + # This mirrors the same suppression on the CUDA plugin EP target. + target_compile_options(onnxruntime_providers_webgpu PRIVATE + $<$,$>:-Wno-maybe-uninitialized> + ) endif() set_target_properties(onnxruntime_providers_webgpu PROPERTIES CXX_STANDARD_REQUIRED ON) From 754e025ddc1168f37f3a7b3bd0948316ed3620b1 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Wed, 26 Aug 2026 13:20:13 -0700 Subject: [PATCH 015/107] Add a work log for the static plugin EP branch Scratch handoff notes covering build configurations, CI dispatch and log retrieval, the GCC -Werror=maybe-uninitialized behavior of the header-only EP adapters, and what still needs validating (notably operator tests on a real GPU). To be removed before the PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../static_plugin_ep_work_log.md | 302 ++++++++++++++++++ 1 file changed, 302 insertions(+) create mode 100644 docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md new file mode 100644 index 0000000000000..48c9fc06a252e --- /dev/null +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md @@ -0,0 +1,302 @@ +# Static plugin EP registration — work log + +Working notes for the branch `edgchen1/webgpu_static_plugin`. This is a scratch handoff document, not a +design document: the design and its rationale live in +[`static_plugin_ep_registration_design.md`](./static_plugin_ep_registration_design.md), which is the +authoritative reference for decisions D1–D7. **Delete this file before opening the PR.** + +The purpose of this log is to let someone pick the work up on a different machine — in particular a machine +with a real GPU, which the original development machine did not have. + +## Goal + +Make the statically linked WebGPU EP build go through the **plugin EP** path rather than the internal-EP path, +via a generic "static plugin EP" registration mechanism in ORT core. The same EP sources +(`onnxruntime/core/providers/webgpu/ep/*`) then serve both a dynamically loaded plugin DLL and a build that is +statically linked into the host. The latter is what unblocks ORT Web / Emscripten, where shared library modules +are not possible. + +Whether the EP is dynamically or statically **linked** is deliberately separate from whether it is a **plugin**. +Both linkage modes are supported. + +## Status + +The implementation is complete and committed. CI coverage was added for the new configuration, which had none. + +| Area | State | +| --- | --- | +| Implementation | Done | +| Windows build + GPU tests (`--use_webgpu static_plugin`) | Green in CI | +| Shared library plugin build (`--use_webgpu shared_lib`) | Green in CI, no regression from D2/D7 | +| CUDA plugin EP (shares `include/onnxruntime/ep/*`) | Green in CI | +| Linux/GCC build (`--use_webgpu static_plugin`) | Fix pushed, awaiting confirmation — see [GCC](#gcc-and-werrormaybe-uninitialized) | +| Minimal build | Not yet exercised — see [Open items](#open-items) | +| Emscripten / ORT Web | Not yet exercised — see [Open items](#open-items) | +| PR | Not opened yet | + +## Branch + +- Branch: `edgchen1/webgpu_static_plugin`, pushed to `git@github.com:microsoft/onnxruntime.git` (not a fork — + this matters, because the self-hosted 1ES CI pools are not available to forks). +- Forked from `main` at `fab4691797560a22092d9bbe8cec540b5413724b`. +- The earlier commits on the branch (`dff9cc42bf` .. `800ad881ef`) are the WebGPU extraction planning documents + and are unrelated to this workstream's code. The code work starts at `2754674229`. + +| Commit | Summary | +| --- | --- | +| `2754674229` | Support registering statically linked plugin EPs, and build WebGPU EP that way | +| `32775c305a` | Add CI jobs for the WebGPU static plugin EP build | +| `49855f6ef7` | Fix GCC maybe-uninitialized errors in the EP adapter attribute accessors | +| `d47f00a0a6` | Update dynamic plugin EP test config parsing tests | +| `1980e901ec` | Record CI coverage in the static plugin EP registration design doc | + +## What the change consists of + +Read the design doc first. The short version, keyed to the decisions: + +- **D1** — A link-time registry in ORT core (`onnxruntime/core/session/plugin_ep/ep_static_plugins.{h,cc}`), + hand-written and `#if`-guarded, mirroring `EpLibraryInternal::CreateInternalEps`. Code generation from CMake + and a new public C API were both considered and rejected. +- **D2** — `ORT_PLUGIN_EP_ENTRY_POINT_PREFIX` in `onnxruntime/core/providers/webgpu/ep/api.cc`. The shared + library build must keep exporting **unprefixed** `CreateEpFactories` / `ReleaseEpFactory`, because + `EpLibraryPlugin::Load` looks those up by name. The static build emits `WebGpu_`-prefixed symbols instead. +- **D3** — No new ABI hook for process-global teardown. `ShutdownProtobufLibrary()` is compiled out unless + `ORT_PLUGIN_EP_OWNS_PROCESS_GLOBALS`. The legacy WebGPU cleanup block in `ort_env.cc` must stay for the + default internal-EP build. +- **D4** — CMake option `onnxruntime_WEBGPU_STATIC_PLUGIN` plus the derived + `onnxruntime_WEBGPU_LINKED_INTO_HOST`; surfaced as `--use_webgpu static_plugin`. +- **D5** — `ep_library_path` became optional in the dynamic plugin EP test infrastructure's + `InitializationConfig`, so a statically registered EP can be selected without a path to load. +- **D6** — Publish-then-register in `ort_env.cc`, and `OrtEnv::m_` became a `std::recursive_mutex`. Registration + reenters the environment (`GetSupportedDevices()` → `Api().ep.GetEnvConfigEntries()` → + `OrtEnv::TryGetInstance()`), which self-deadlocked with a plain mutex. +- **D7** — `ORT_PLUGIN_EP_STATICALLY_LINKED` disables the manual C++ API init in the shared + `include/onnxruntime/ep/api.h`. Do not try to force `ORT_API_MANUAL_INIT` on for everyone: MSVC's + `#pragma detect_mismatch` requires whole-binary agreement and doing so produced roughly 50 `LNK2038` errors. + +Files touched outside `docs/`: + +``` +.github/workflows/linux_webgpu.yml +.github/workflows/windows_webgpu.yml +cmake/CMakeLists.txt +cmake/onnxruntime.cmake +cmake/onnxruntime_providers_webgpu.cmake +cmake/onnxruntime_python.cmake +cmake/onnxruntime_unittests.cmake +include/onnxruntime/core/session/environment.h +include/onnxruntime/ep/adapter/op_kernel_info.h +include/onnxruntime/ep/api.h +onnxruntime/core/providers/webgpu/ep/api.cc +onnxruntime/core/providers/webgpu/math/{gemm,softmax,top_k}.h +onnxruntime/core/providers/webgpu/tensor/cast.h +onnxruntime/core/session/environment.cc +onnxruntime/core/session/ort_env.{h,cc} +onnxruntime/core/session/plugin_ep/ep_library_plugin.cc +onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.{h,cc} (new) +onnxruntime/core/session/plugin_ep/ep_library_static_plugin.{h,cc} (new) +onnxruntime/core/session/plugin_ep/ep_static_plugins.{h,cc} (new) +onnxruntime/test/framework/dynamic_plugin_ep_test.cc +onnxruntime/test/framework/inference_session_test.cc +onnxruntime/test/unittest_main/test_main.cc +onnxruntime/test/unittest_util/test_dynamic_plugin_ep.{h,cc} +tools/ci_build/{build.py,build_args.py} +``` + +## Building + +Three WebGPU configurations exist and it is worth being able to build at least two of them, because several +bugs in this work only appeared in one of them: + +``` +python tools/ci_build/build.py ... --use_webgpu # internal EP (the default, pre-existing path) +python tools/ci_build/build.py ... --use_webgpu shared_lib # plugin EP in its own shared library +python tools/ci_build/build.py ... --use_webgpu static_plugin # plugin EP linked into the host <-- the new one +``` + +`--use_webgpu static_plugin` maps to `-Donnxruntime_USE_EP_API_ADAPTERS=ON +-Donnxruntime_WEBGPU_STATIC_PLUGIN=ON` (see `tools/ci_build/build.py`, the `--use_webgpu` handling). Both +options default to `OFF`. + +`onnxruntime_USE_EP_API_ADAPTERS=ON` is **incompatible** with `onnxruntime_BUILD_DAWN_SHARED_LIBRARY=ON` +(`cmake/CMakeLists.txt` raises a `FATAL_ERROR`). The internal-EP WebGPU CI job sets the latter; the plugin jobs +must not. + +### Windows gotchas + +These cost real time on the original machine and are likely to recur: + +- **`MAX_PATH`.** Building WebGPU under a long path such as `\build\Windows` makes Dawn's DXC build fail + with `MSB3491`. Build into a short directory instead — `C:\b` and `C:\b2` were used for the `static_plugin` + and `shared_lib` configurations respectively. With the Visual Studio generator the binaries land in + `\\\`. +- **CMake may not be on `PATH`.** On a Visual Studio install it is at + `C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin`. +- Incremental rebuild of just the EP is much faster than a full build: + `cmake --build --config Debug --target onnxruntime_providers_webgpu --parallel`. + +### Verifying the D2 export contract + +For the **shared library** configuration, confirm the exported entry points are still unprefixed, since +`EpLibraryPlugin::Load` resolves them by name: + +``` +dumpbin /exports \\\onnxruntime_providers_webgpu.dll +``` + +Expect exactly `CreateEpFactories` and `ReleaseEpFactory`. The `static_plugin` build should instead produce +`WebGpu_`-prefixed symbols and no DLL. + +## Testing, and why a GPU machine is needed + +Relevant tests: + +- `onnxruntime_test_all --gtest_filter=InferenceSessionTests.WebGpu*` — covers registration, device + enumeration, and session creation end to end through the public V2 API. +- `onnxruntime_provider_test` — WebGPU EP kernel tests. + +The original development machine was a **Hyper-V VM with no real GPU**. On such a host WebGPU only surfaces a +*virtual* `OrtEpDevice`, and any session that is not compile-only is rejected up front with +*"WebGPU EP was selected on a virtual GPU device…"*. So locally only these two pass: + +- `InferenceSessionTests.WebGpuVirtualDeviceCompileOnlyEndToEnd` +- `InferenceSessionTests.WebGpuVirtualDeviceRejectedWithoutCompileOnly` + +and `WebGpuCompileOnlySkipsFinalization` and `TestStrictShapeInference` fail. This was proven to be a property +of the host and not of static linking, by building `--use_webgpu shared_lib` on the same machine and observing +the identical pass/fail set. + +**Consequence for whoever picks this up:** operator execution cannot be validated on a GPU-less machine at all. +Two ways to get real coverage: + +1. **Use CI.** The new `webgpu_static_plugin_build_x64_RelWithDebInfo` job in `.github/workflows/windows_webgpu.yml` + runs on the `onnxruntime-github-Win2022-GPU-A10` pool, which is the only GPU-equipped WebGPU pool, and it + both builds and tests. It has passed. This is currently the primary source of operator-level truth. +2. **Run locally on a real GPU.** Nothing in the change is GPU-vendor specific; a normal Windows machine with a + D3D12 or Vulkan capable GPU is enough. + +There is also a `webgpu-local-testing` skill in `.agents/skills/` describing how to run WebGPU tests on Linux +without a GPU using Mesa lavapipe (software Vulkan). Note its stated limitation: any graph containing `MatMul` +crashes lavapipe, so it only validates host-side enforcement and shape logic. + +### Not covered by any test yet + +Re-registration — register, run, unregister, re-register — is untested and is not reachable from the existing +test binaries, because `OrtEnv` is a refcounted process singleton that the unit test main pins for the lifetime +of the process. + +## CI + +No pre-existing leg built the `static_plugin` configuration, since both CMake options default to `OFF`. Every +WebGPU job was either internal EP (`--use_webgpu`) or shared library plugin (`--use_webgpu shared_lib`). Two +jobs were added: + +- `webgpu_static_plugin_build_x64_RelWithDebInfo` in `.github/workflows/windows_webgpu.yml` — builds **and + tests** on the A10 GPU pool. It deliberately omits `onnxruntime_BUILD_DAWN_SHARED_LIBRARY=ON` (incompatible, + see above) and deliberately drops the `--disable_rtti` / `--enable_lto` flags that the shared library job + carries, since those describe how the plugin ships as a DLL and here they would apply to all of ORT. +- `build-linux-webgpu-static-plugin-x64-release` in `.github/workflows/linux_webgpu.yml` — build only, mirroring + its sibling job. This is the only **GCC** coverage of the EP adapters, and the only coverage of the + `cmake/onnxruntime_python.cmake` change, which is why `--build_wheel` is kept on it. + +`linux_webgpu.yml` is build-only by design; its test job is commented out with a pre-existing "currently +failing" TODO. + +### Running CI on this branch + +The workflows only trigger automatically on `push`/`pull_request` for `main`, `rel-*`, and +`plugin-ep-webgpu/rel-*`, so **pushing this branch triggers nothing**. Dispatch manually: + +``` +gh workflow run windows_webgpu.yml --repo microsoft/onnxruntime --ref edgchen1/webgpu_static_plugin +gh workflow run linux_webgpu.yml --repo microsoft/onnxruntime --ref edgchen1/webgpu_static_plugin +gh workflow run windows_cuda_plugin.yml --repo microsoft/onnxruntime --ref edgchen1/webgpu_static_plugin +``` + +`gh workflow run --ref ` reads the workflow file **from that ref**, so jobs added on the branch do run +even though they are not on `main` yet. This was confirmed empirically. + +`windows_cuda_plugin.yml` matters because the CUDA plugin EP shares `include/onnxruntime/ep/*` with WebGPU, so +it is the regression check for the D5 and D7 changes. + +### Retrieving logs + +`gh run view --log-failed` and `gh run view --job --log` both returned **zero bytes** for these runs. Use +the REST API instead: + +``` +gh api repos/microsoft/onnxruntime/actions/runs//jobs --jq '.jobs[] | select(.conclusion=="failure") | "\(.id) \(.name)"' +gh api repos/microsoft/onnxruntime/actions/jobs//logs > log.txt +``` + +When grepping the result, search for `FAILED:` **case-sensitively**. A case-insensitive search for "failed" or +a search for `error:` drowns in benign CMake and Dawn feature-probe output such as +`-- Performing Test ... - Failed`. + +### Run history + +| Run | Workflow | Result | +| --- | --- | --- | +| 32914301463 | ONNX Runtime WebGPU Builds | Success, all 6 jobs including the new one | +| 32914304408 | Linux WebGPU CI | Failure — GCC `maybe-uninitialized`, batch 1 | +| 32914307098 | CUDA Plugin Windows CI | Failure — `ParseInitializationConfigRejectsMissingRequiredFields` | +| 32996643911 | Linux WebGPU CI | Failure — GCC `maybe-uninitialized`, batch 2 | +| 32996646394 | CUDA Plugin Windows CI | Success | + +## GCC and `-Werror=maybe-uninitialized` + +This was the only substantial problem the new CI surfaced, and it will bite again, so it is worth understanding +rather than pattern-matching. + +The EP adapter types are defined **entirely inline in headers**. The in-tree `OpKernelInfo` hides its +accessors behind out-of-line definitions in a `.cc`, so GCC cannot see their failure and exception paths. With +the adapters, it can — and it then warns about values that are only ever read once the call has succeeded. The +warning is a false positive, but the build uses `-Werror`. + +This had never been observed before simply because the EP-adapters path had never been compiled with GCC: the +existing Linux WebGPU CI builds the internal EP, and the shared library plugin builds are Windows/MSVC-only. +The CUDA plugin EP hit the identical problem earlier and solved it with a target-scoped suppression in +`cmake/onnxruntime_providers_cuda_plugin.cmake`. + +Two batches of failures appeared, because ninja stops early and only reports the targets that were already in +flight — **do not assume the first batch is the complete list**. + +- Batch 1, fixed at the source in `49855f6ef7`: uninitialized locals that are passed by pointer and read back, + and the temporaries inside `GetAttrOrDefault` / `GetAttr` in `include/onnxruntime/ep/adapter/op_kernel_info.h`. + Value-initializing them (`T tmp{}`) is correct and worth keeping on its own merits. +- Batch 2: false positives deep inside `absl::InlinedVector`, for `TensorShapeVector` locals in + `tensor/pad.cc` and `tensor/unsqueeze.h` that are always properly constructed. These cannot be fixed at the + source. Handled by applying the same `-Wno-maybe-uninitialized` suppression the CUDA plugin EP uses, scoped + to the `onnxruntime_providers_webgpu` target, GCC only, and only when `onnxruntime_USE_EP_API_ADAPTERS` is on + so that non-adapter builds stay strict. + +## Open items + +1. **Confirm the Linux leg is green** after the `-Wno-maybe-uninitialized` suppression. Dispatch a fresh run + rather than re-running an old one. +2. **Operator-level validation on a real GPU.** The Windows CI job covers this now, but nobody has run the + WebGPU operator tests against the `static_plugin` build on a local GPU machine and compared the results to + the internal-EP build. That comparison is the strongest available evidence that routing the static build + through the plugin path changes no behaviour, and it is the main reason to move this work to a GPU machine. +3. **Minimal build.** `linux_minimal_build.yml` has not been dispatched. It is the interesting one because + `cmake/onnxruntime_session.cmake` excludes `plugin_ep/*.*` from minimal builds, which is why the + registration call in `ort_env.cc` is wrapped in `#if !defined(ORT_MINIMAL_BUILD)`. The guard has been + inspected and looks correct but has not been exercised. Also unknown: whether any shipped ORT Web + configuration uses an extended-minimal build, which would exclude the plugin EP infrastructure entirely. +4. **Emscripten / ORT Web.** This is the motivating use case for the whole change and has not been built. + `web.yml` exists and supports `workflow_dispatch`. +5. **Follow-up cleanup** listed at the end of the design doc, including removing the WebGPU special cases that + only exist while it is still an "internal" EP. +6. **Open the PR.** + +## Environment notes + +Specific to the original machine, but the class of problem generalises. + +- The working copy `C:\source\ort-webgpu-static-plugin` is a **linked git worktree** of `C:\source\onnxruntime`. + `.git` is therefore a *file*, not a directory. Do not write scratch files into it. +- Linting is `lintrunner`, driven by the `ort-lint` skill. `lintrunner -a` can report "ok No lint issues" and + "Successfully applied all patches" in the same run, so always re-check `git status` afterwards. +- The repository's own pre-commit hook lives in `.githooks/pre-commit` (this is not the `pre-commit` framework) + and is enabled with `git config core.hooksPath .githooks`. It **silently does nothing if `lintrunner` is not + on `PATH`**, so activate the virtual environment before committing or the commit goes through unlinted. From 35e6f65d5224b57ff3d56e76c13315376b2e63ae Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Wed, 26 Aug 2026 15:22:07 -0700 Subject: [PATCH 016/107] Record real-GPU validation results in the static plugin EP work log Built the static_plugin and internal-EP WebGPU configurations from the same tree on a machine with a real GPU and compared them. onnxruntime_provider_test: zero failures in both, and zero status differences across the 5894 tests common to the two builds. onnxruntime_test_all: 1965 passed in both, no differences. This closes open item 2, the operator-level validation that could not be done on the original GPU-less development machine. Also records the two categories of test that legitimately differ (white-box WebGPU tests excluded from adapter builds, and the deliberately paired virtual-device tests), the third GCC -Werror batch seen in run 33009901327, and the finding that the MAX_PATH build directory restriction is specific to MSBuild rather than to the path length itself. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../static_plugin_ep_work_log.md | 92 +++++++++++++++++-- 1 file changed, 82 insertions(+), 10 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md index 48c9fc06a252e..f390094c2fe3b 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md @@ -29,7 +29,8 @@ The implementation is complete and committed. CI coverage was added for the new | Windows build + GPU tests (`--use_webgpu static_plugin`) | Green in CI | | Shared library plugin build (`--use_webgpu shared_lib`) | Green in CI, no regression from D2/D7 | | CUDA plugin EP (shares `include/onnxruntime/ep/*`) | Green in CI | -| Linux/GCC build (`--use_webgpu static_plugin`) | Fix pushed, awaiting confirmation — see [GCC](#gcc-and-werrormaybe-uninitialized) | +| Operator-level validation on a real GPU | **Done — no behavioural difference**, see [Testing](#testing) | +| Linux/GCC build (`--use_webgpu static_plugin`) | **Red** — third `-Werror` batch (`array-bounds`), see [GCC](#gcc-and-werror) | | Minimal build | Not yet exercised — see [Open items](#open-items) | | Emscripten / ORT Web | Not yet exercised — see [Open items](#open-items) | | PR | Not opened yet | @@ -130,6 +131,11 @@ These cost real time on the original machine and are likely to recur: with `MSB3491`. Build into a short directory instead — `C:\b` and `C:\b2` were used for the `static_plugin` and `shared_lib` configurations respectively. With the Visual Studio generator the binaries land in `\\\`. + + Refinement from a later machine: this is really an **MSBuild** problem. With `--cmake_generator Ninja` and + the registry's `LongPathsEnabled=1`, building at `D:\source\onnxruntime_4\build\sp` worked fine, so an + in-repo build directory is usable if you avoid MSBuild. Note that with Ninja the binaries land in + `\\`, one level, not two. - **CMake may not be on `PATH`.** On a Visual Studio install it is at `C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin`. - Incremental rebuild of just the EP is much faster than a full build: @@ -147,7 +153,7 @@ dumpbin /exports \\\onnxruntime_providers_webgpu.dll Expect exactly `CreateEpFactories` and `ReleaseEpFactory`. The `static_plugin` build should instead produce `WebGpu_`-prefixed symbols and no DLL. -## Testing, and why a GPU machine is needed +## Testing Relevant tests: @@ -175,6 +181,54 @@ Two ways to get real coverage: 2. **Run locally on a real GPU.** Nothing in the change is GPU-vendor specific; a normal Windows machine with a D3D12 or Vulkan capable GPU is enough. +### Results on a real GPU + +This has now been done, on a machine with an NVIDIA RTX 5060 Ti. Both configurations were built from the same +tree, RelWithDebInfo, Ninja, differing **only** in the EP path: + +``` +python tools\ci_build\build.py --config RelWithDebInfo --build_dir --skip_submodule_sync --parallel \ + --cmake_generator Ninja --use_webgpu [static_plugin] --update --build --skip_tests \ + --cmake_extra_defines onnxruntime_BUILD_UNIT_TESTS=ON \ + onnxruntime_ENABLE_DAWN_BACKEND_D3D12=1 onnxruntime_ENABLE_DAWN_BACKEND_VULKAN=1 +``` + +`onnxruntime_provider_test`, with `ONNXRUNTIME_TEST_GPU_DEVICE_ID=0`: + +| | static_plugin | internal EP | +| --- | --- | --- | +| Total | 5894 | 5938 | +| Passed | 5775 | 5803 | +| Failed | 0 | 0 | +| Skipped | 119 | 135 | + +**Across the 5894 tests common to both builds there were zero status differences.** This is the evidence that +routing the static build through the plugin path changes no operator behaviour. + +`onnxruntime_test_all`: 1965 passed on both, no differences. All three `InferenceSessionTests.WebGpu*` pass on +the `static_plugin` build, including `WebGpuCompileOnlySkipsFinalization` — confirming the earlier failure of +that test was a property of the GPU-less host, as suspected. + +Two categories of test legitimately differ between the builds; neither is a regression: + +- **44 tests exist only in the internal-EP build.** 16 are `DISABLED_` and never run. The other 28 are + white-box tests that include internal WebGPU EP headers (`WebGpuContextTest`, `ActivationCacheKeyTest`, + `MatMul2BitsWebGpu`, `HardSwish_WebGPU`); `cmake/onnxruntime_unittests.cmake` (the + `onnxruntime_USE_WEBGPU AND NOT onnxruntime_USE_EP_API_ADAPTERS` guard) excludes `test/providers/webgpu/*` + from adapter builds entirely. Of these, `HardSwish_WebGPU` and `MatMul2BitsWebGpu` (12 tests) are real + operator coverage that exists only on the internal path — a known coverage gap of the plugin path, worth + either documenting in the design doc or closing by porting them to EP-agnostic tests. +- **The `InferenceSessionTests.WebGpu*` virtual-device tests are deliberately paired, not shared.** The + internal build has `WebGpuEpFactoryVirtualDevice`, `WebGpuEpFactoryRejectsVirtualDeviceWithoutCompileOnly` + and `WebGpuCompileOnlyUsesNoOpAllocator`; the plugin build has `WebGpuVirtualDeviceCompileOnlyEndToEnd` and + `WebGpuVirtualDeviceRejectedWithoutCompileOnly`, which `inference_session_test.cc` documents in-source as the + plugin-build counterparts. Only `WebGpuCompileOnlyUsesNoOpAllocator` has no counterpart, because it asserts + an internal allocator *type* via `dynamic_cast`, which is inherently unobservable across the ABI boundary. + +One caveat for whoever repeats this: do **not** run the two `onnxruntime_test_all` binaries concurrently. +`PathValidationTest`'s symlink tests collide over a shared temp directory and produce a spurious, and +non-deterministic, mix of pass/fail/skip. Run sequentially: all 31 pass on both builds. + There is also a `webgpu-local-testing` skill in `.agents/skills/` describing how to run WebGPU tests on Linux without a GPU using Mesa lavapipe (software Vulkan). Note its stated limitation: any graph containing `MatMul` crashes lavapipe, so it only validates host-side enforcement and shape logic. @@ -242,8 +296,9 @@ a search for `error:` drowns in benign CMake and Dawn feature-probe output such | 32914307098 | CUDA Plugin Windows CI | Failure — `ParseInitializationConfigRejectsMissingRequiredFields` | | 32996643911 | Linux WebGPU CI | Failure — GCC `maybe-uninitialized`, batch 2 | | 32996646394 | CUDA Plugin Windows CI | Success | +| 33009901327 | Linux WebGPU CI | Failure — GCC `array-bounds`, batch 3. Internal-EP job on the same run passed. | -## GCC and `-Werror=maybe-uninitialized` +## GCC and `-Werror` This was the only substantial problem the new CI surfaced, and it will bite again, so it is worth understanding rather than pattern-matching. @@ -259,7 +314,9 @@ The CUDA plugin EP hit the identical problem earlier and solved it with a target `cmake/onnxruntime_providers_cuda_plugin.cmake`. Two batches of failures appeared, because ninja stops early and only reports the targets that were already in -flight — **do not assume the first batch is the complete list**. +flight — **do not assume the first batch is the complete list**. A third batch, of a different warning class, +appeared later still. The lesson is the general one: **each CI round trip reveals only one batch**, so prefer a +local GCC build (or a throwaway dispatch with `--compile_no_warning_as_error`) to enumerate all of them at once. - Batch 1, fixed at the source in `49855f6ef7`: uninitialized locals that are passed by pointer and read back, and the temporaries inside `GetAttrOrDefault` / `GetAttr` in `include/onnxruntime/ep/adapter/op_kernel_info.h`. @@ -269,15 +326,30 @@ flight — **do not assume the first batch is the complete list**. source. Handled by applying the same `-Wno-maybe-uninitialized` suppression the CUDA plugin EP uses, scoped to the `onnxruntime_providers_webgpu` target, GCC only, and only when `onnxruntime_USE_EP_API_ADAPTERS` is on so that non-adapter builds stay strict. +- Batch 3, in run `33009901327`, **still unfixed**: `-Werror=array-bounds`, a *different* warning class, at + `onnxruntime/contrib_ops/webgpu/quantization/matmul_nbits_mlp.cc:47` in `EmitGateActivationExpr`: + + ``` + ‘void* __builtin_memcpy(...)’ forming offset [32, 48] is out of the bounds [0, 32] + of object ‘’ with type ‘std::__cxx11::basic_string’ + ``` + + GCC 14 at `-O3`, inlining the `std::string{gate_var} + " * (one / (one + exp(-" + std::string{gate_var} + ")))"` + temporary-move concat chain through `std::operator+(basic_string&&, const char*)`. A false positive. Note that + the batch-2 `-Wno-maybe-uninitialized` suppression **is** working — it is present on the failing command line + and the batch-2 files compiled clean. The file is pre-existing (added by PR #28280, `88ca23fcd5`), not authored + by this branch; the adapters PCH pulls in `include/onnxruntime/ep/adapters.h`, which changes inlining and is + why only the adapters config trips it. Fix not yet chosen: rewrite at the source (e.g. ORT's `MakeString`, or + reserve + append) versus extending the existing target-scoped suppression with `-Wno-array-bounds`. ## Open items -1. **Confirm the Linux leg is green** after the `-Wno-maybe-uninitialized` suppression. Dispatch a fresh run - rather than re-running an old one. -2. **Operator-level validation on a real GPU.** The Windows CI job covers this now, but nobody has run the - WebGPU operator tests against the `static_plugin` build on a local GPU machine and compared the results to - the internal-EP build. That comparison is the strongest available evidence that routing the static build - through the plugin path changes no behaviour, and it is the main reason to move this work to a GPU machine. +1. **Fix the Linux leg.** It is currently red on a third `-Werror` batch (`array-bounds`); see + [GCC](#gcc-and-werror). Enumerate the *full* remaining warning list with a local GCC build rather than + discovering one batch per CI round trip. Dispatch a fresh run rather than re-running an old one. +2. ~~**Operator-level validation on a real GPU.**~~ **Done** — see [Results on a real GPU](#results-on-a-real-gpu). + No behavioural difference. Remaining sub-item: decide whether to close the 12-test + `HardSwish_WebGPU` / `MatMul2BitsWebGpu` coverage gap on the plugin path, or document it as known. 3. **Minimal build.** `linux_minimal_build.yml` has not been dispatched. It is the interesting one because `cmake/onnxruntime_session.cmake` excludes `plugin_ep/*.*` from minimal builds, which is why the registration call in `ort_env.cc` is wrapped in `#if !defined(ORT_MINIMAL_BUILD)`. The guard has been From a110948b1d5237ee238e64237d4729278b08f59e Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Wed, 26 Aug 2026 16:10:53 -0700 Subject: [PATCH 017/107] Record GPU validation and the plugin-path test coverage gap in the design doc The work log is deleted before the PR, so these findings need to live in the design document instead. Records that operator-level validation on a real GPU is now done and showed zero status differences across the 5894 provider tests common to the static_plugin and internal-EP builds. Also documents the known coverage difference: onnxruntime_unittests.cmake excludes test/providers/webgpu/* from adapter builds because those tests include internal EP headers, which leaves HardSwish_WebGPU and MatMul2BitsWebGpu (12 operator tests) running only on the internal path. Explains why the virtual-device tests are deliberately paired rather than shared, and generalises the GCC -Werror guidance now that several successive batches have appeared. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../static_plugin_ep_registration_design.md | 53 ++++++++++++++++--- 1 file changed, 47 insertions(+), 6 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md index 71daa7d207048..9547790aca177 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -273,13 +273,43 @@ Verified on Windows with `--use_webgpu static_plugin`, Debug: `InferenceSessionTests.WebGpuVirtualDeviceRejectedWithoutCompileOnly` pass, which covers session creation and EP instantiation end to end through the public V2 API on the static plugin path. -Not yet verified locally: operator execution. The development machine is a virtual machine with no real GPU, so only -the virtual WebGPU device is surfaced and non-compile-only sessions are rejected up front. This was confirmed to be a -property of the host rather than of static linking by building the shared library plugin configuration -(`--use_webgpu shared_lib`) on the same host and running the same tests: it passes and fails exactly the same set -(`WebGpuVirtualDeviceCompileOnlyEndToEnd` and `WebGpuVirtualDeviceRejectedWithoutCompileOnly` pass; +Operator execution was initially not verifiable locally: the original development machine was a virtual machine with +no real GPU, so only the virtual WebGPU device was surfaced and non-compile-only sessions were rejected up front. This +was confirmed to be a property of the host rather than of static linking by building the shared library plugin +configuration (`--use_webgpu shared_lib`) on the same host and running the same tests: it passed and failed exactly +the same set (`WebGpuVirtualDeviceCompileOnlyEndToEnd` and `WebGpuVirtualDeviceRejectedWithoutCompileOnly` pass; `WebGpuCompileOnlySkipsFinalization` and `TestStrictShapeInference` fail with the identical virtual-device rejection -message). Operator coverage on either plugin path requires a host with a GPU. +message). + +This gap has since been closed on a machine with a real GPU (NVIDIA RTX 5060 Ti). The `static_plugin` and internal-EP +configurations were built from the same tree, RelWithDebInfo, differing only in the EP path, and their test results +compared: + +- `onnxruntime_provider_test`: 0 failures in both. **Across the 5894 tests common to both builds there were zero + status differences.** +- `onnxruntime_test_all`: 1965 passed in both, no differences. All three `InferenceSessionTests.WebGpu*` pass on the + static plugin build, including `WebGpuCompileOnlySkipsFinalization`, confirming its earlier failure was indeed a + property of the GPU-less host. + +### Known test coverage difference on the plugin path + +`cmake/onnxruntime_unittests.cmake` gates the `test/providers/webgpu/*` sources on +`onnxruntime_USE_WEBGPU AND NOT onnxruntime_USE_EP_API_ADAPTERS`, so those tests are **absent** from any adapter +build, including `static_plugin`. They are white-box tests that include internal WebGPU EP headers and therefore +cannot compile against the adapter boundary by construction. + +Of the 44 tests this excludes, 16 are `DISABLED_` and never run, and most of the rest assert on EP internals +(`WebGpuContextTest`, `ActivationCacheKeyTest`). However, 12 of them — `HardSwish_WebGPU` and `MatMul2BitsWebGpu` — +are genuine operator tests, and they currently run **only** on the internal EP path. This is a known coverage gap of +the plugin path. Closing it would mean rewriting them as EP-agnostic operator tests so both paths execute them; that +is deliberately left out of this change to keep it focused. + +Separately, the `InferenceSessionTests.WebGpu*` virtual-device tests are deliberately *paired* rather than shared +between the two builds, since each drives a different factory. `WebGpuVirtualDeviceCompileOnlyEndToEnd` and +`WebGpuVirtualDeviceRejectedWithoutCompileOnly` are the plugin-build counterparts of `WebGpuEpFactoryVirtualDevice` +and `WebGpuEpFactoryRejectsVirtualDeviceWithoutCompileOnly`. Only `WebGpuCompileOnlyUsesNoOpAllocator` has no +counterpart, because it asserts an internal allocator *type* through `dynamic_cast`, which is inherently +unobservable across the ABI boundary. The shared library plugin configuration was also re-verified after the D7 change, since `include/onnxruntime/ep/api.h` is shared with the CUDA plugin EP. It builds, and `onnxruntime_providers_webgpu.dll` still exports exactly the @@ -301,6 +331,17 @@ attribute accessors inline in the header, so GCC can see their failure and excep `OpKernelInfo` hides them behind an out-of-line definition. Any value that is only read when the attribute lookup succeeded must be value-initialized. +More generally, this job is the first time the EP adapters have ever been compiled with GCC, and it has surfaced +several successive batches of `-Werror` diagnostics — including false positives inside `absl::InlinedVector` and, in +optimized builds, `-Werror=array-bounds` on `std::string` concatenation chains that the adapter headers cause to be +inlined more aggressively. Because ninja stops at the first failure and reports only the targets already in flight, +**each CI run reveals just one batch**. When touching this configuration, enumerate the full set locally in a single +pass — an AlmaLinux 8 container with `gcc-toolset-14` reproduces the CI compiler exactly — rather than discovering +them one CI round trip at a time. Fixes that are correct on their own merits are applied at the source; irreducible +false positives use a suppression scoped to the `onnxruntime_providers_webgpu` target, GCC only, and conditional on +`onnxruntime_USE_EP_API_ADAPTERS`, so non-adapter builds stay strict. This mirrors what the CUDA plugin EP does in +`cmake/onnxruntime_providers_cuda_plugin.cmake`. + ## Resolution of workstream open questions > Should static factories be registered before environment creation or through environment construction options? From 8d87c119727b284ada249b65aec8a8705a289d1b Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Wed, 26 Aug 2026 16:43:23 -0700 Subject: [PATCH 018/107] Avoid GCC 14 -Warray-bounds false positive in EmitGateActivationExpr When onnxruntime_providers_webgpu is built with the EP adapter headers in its precompiled header, GCC 14 at -O3 inlines the std::string move-concat chain in EmitGateActivationExpr and reports a false positive -Warray-bounds, which -Werror turns into a build failure on the Linux static_plugin CI leg. Build the expression with ORT's MakeString instead. This is a single-site source fix and keeps -Warray-bounds active for the rest of the provider, unlike a target-scoped -Wno-array-bounds. Verified with GCC 14.2.1 in an AlmaLinux 8 gcc-toolset-14 container: the unfixed source still errors under -Werror through the same harness, the fixed source compiles clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../contrib_ops/webgpu/quantization/matmul_nbits_mlp.cc | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/onnxruntime/contrib_ops/webgpu/quantization/matmul_nbits_mlp.cc b/onnxruntime/contrib_ops/webgpu/quantization/matmul_nbits_mlp.cc index dd106c66f0c4e..6d8fac7dbaf85 100644 --- a/onnxruntime/contrib_ops/webgpu/quantization/matmul_nbits_mlp.cc +++ b/onnxruntime/contrib_ops/webgpu/quantization/matmul_nbits_mlp.cc @@ -11,6 +11,7 @@ #include "contrib_ops/webgpu/quantization/dp4a_matmul_nbits.h" #include "contrib_ops/webgpu/bert/skip_layer_norm.h" #include "contrib_ops/webgpu/webgpu_contrib_kernels.h" +#include "core/common/make_string.h" #include "core/providers/cpu/math/matmul_helper.h" #include "core/providers/webgpu/nn/layer_norm.h" #include "core/providers/webgpu/shader_helper.h" @@ -44,7 +45,10 @@ std::string EmitGateActivationExpr(MlpActivationKind kind, std::string_view gate switch (kind) { case MlpActivationKind::Silu: // SiLU(x) = x * sigmoid(x) - return std::string{gate_var} + " * (one / (one + exp(-" + std::string{gate_var} + ")))"; + // Built with MakeString rather than a std::string concatenation chain: GCC 14 at -O3 reports a false + // positive -Warray-bounds through the inlined basic_string move constructor when this translation unit + // is compiled with the EP adapter headers in the precompiled header. + return MakeString(gate_var, " * (one / (one + exp(-", gate_var, ")))"); } ORT_THROW("MatMulNBitsMlp: unhandled MlpActivationKind ", static_cast(kind)); } From ef68884a952145f9da4dc04f2b2c8f9b6aed3a32 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Wed, 26 Aug 2026 16:43:39 -0700 Subject: [PATCH 019/107] Record the complete GCC 14 warning inventory and the local reproduction recipe Closes the batch-at-a-time discovery loop: a single container build with --compile_no_warning_as_error shows that of ~340 warnings in the static_plugin build, exactly one belongs to ORT source. Documents the AlmaLinux 8 / gcc-toolset-14 reproduction, the single-object -Werror recheck, and the need for a negative control. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../static_plugin_ep_registration_design.md | 11 ++++ .../static_plugin_ep_work_log.md | 66 +++++++++++++++++-- 2 files changed, 70 insertions(+), 7 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md index 9547790aca177..d62c7456c454c 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -342,6 +342,17 @@ false positives use a suppression scoped to the `onnxruntime_providers_webgpu` t `onnxruntime_USE_EP_API_ADAPTERS`, so non-adapter builds stay strict. This mirrors what the CUDA plugin EP does in `cmake/onnxruntime_providers_cuda_plugin.cmake`. +That full local enumeration has since been done, and it is worth recording what it found: of the ~340 warnings GCC 14 +emits across the whole `static_plugin` build, **exactly one is attributable to ORT's own source** — the +`-Warray-bounds` false positive in `contrib_ops/webgpu/quantization/matmul_nbits_mlp.cc`, fixed by building the +expression with `MakeString` instead of a `std::string` concat chain. Everything else lives in Dawn, ONNX, +flatbuffers, or the telemetry dependency, in targets that do not use `-Werror`. The practical lesson is that the +adapters do not make ORT's code broadly warning-prone; they perturb inlining at a small number of specific sites, +so a targeted source fix is almost always available and preferable to a blanket target-wide suppression. + +When validating such a fix locally, always rebuild the *unfixed* source through the same harness first and confirm it +still errors. A harness that silently fails to exercise the warning is indistinguishable from a successful fix. + ## Resolution of workstream open questions > Should static factories be registered before environment creation or through environment construction options? diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md index f390094c2fe3b..990cbedc3458e 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md @@ -30,7 +30,7 @@ The implementation is complete and committed. CI coverage was added for the new | Shared library plugin build (`--use_webgpu shared_lib`) | Green in CI, no regression from D2/D7 | | CUDA plugin EP (shares `include/onnxruntime/ep/*`) | Green in CI | | Operator-level validation on a real GPU | **Done — no behavioural difference**, see [Testing](#testing) | -| Linux/GCC build (`--use_webgpu static_plugin`) | **Red** — third `-Werror` batch (`array-bounds`), see [GCC](#gcc-and-werror) | +| Linux/GCC build (`--use_webgpu static_plugin`) | `array-bounds` fixed at the source; verified locally under GCC 14, **not yet confirmed in CI** — see [GCC](#gcc-and-werror) | | Minimal build | Not yet exercised — see [Open items](#open-items) | | Emscripten / ORT Web | Not yet exercised — see [Open items](#open-items) | | PR | Not opened yet | @@ -326,7 +326,7 @@ local GCC build (or a throwaway dispatch with `--compile_no_warning_as_error`) t source. Handled by applying the same `-Wno-maybe-uninitialized` suppression the CUDA plugin EP uses, scoped to the `onnxruntime_providers_webgpu` target, GCC only, and only when `onnxruntime_USE_EP_API_ADAPTERS` is on so that non-adapter builds stay strict. -- Batch 3, in run `33009901327`, **still unfixed**: `-Werror=array-bounds`, a *different* warning class, at +- Batch 3, in run `33009901327`, **fixed at the source**: `-Werror=array-bounds`, a *different* warning class, at `onnxruntime/contrib_ops/webgpu/quantization/matmul_nbits_mlp.cc:47` in `EmitGateActivationExpr`: ``` @@ -339,14 +339,66 @@ local GCC build (or a throwaway dispatch with `--compile_no_warning_as_error`) t the batch-2 `-Wno-maybe-uninitialized` suppression **is** working — it is present on the failing command line and the batch-2 files compiled clean. The file is pre-existing (added by PR #28280, `88ca23fcd5`), not authored by this branch; the adapters PCH pulls in `include/onnxruntime/ep/adapters.h`, which changes inlining and is - why only the adapters config trips it. Fix not yet chosen: rewrite at the source (e.g. ORT's `MakeString`, or - reserve + append) versus extending the existing target-scoped suppression with `-Wno-array-bounds`. + why only the adapters config trips it. Fixed by building the expression with ORT's `MakeString` instead of a + `std::string` concat chain — a single-site source fix, preferred over extending the target-scoped suppression + with `-Wno-array-bounds`, which would have blinded the whole provider to a genuinely useful warning. + +### The complete GCC 14 warning inventory + +The batch-at-a-time discovery loop was finally closed by enumerating **every** warning in one pass: a full +container build with `--compile_no_warning_as_error`, then grouping the log by warning class. The result: + +| Warning class | Count | Where | +| --- | --- | --- | +| `-Wattributes` | 276 | Dawn generated code (`dawn_proc.cpp`) | +| `-Wunused-parameter` | 55 | ONNX (`onnx/defs/**`) | +| `-Wdangling-pointer=` | 3 | Dawn / Tint | +| `-Wchanges-meaning` | 3 | Dawn (`SystemEvent.h`) | +| `-Wstringop-overflow=` | 2 | flatbuffers (`reflection.cpp`) | +| `-Warray-bounds=` | **1** | **ORT — `matmul_nbits_mlp.cc:47`** | +| `-Wstringop-overread` | 1 | telemetry (`sqlite3_retail.c`) | + +**Exactly one warning in the whole build is attributable to ORT's own source**, and it is the batch-3 failure. +Everything else is third-party, in targets that do not compile with `-Werror`. So there is no batch 4: fixing +this one site is expected to turn the leg green. + +### Reproducing the CI compiler locally + +Worth the setup cost — it turns a ~28 minute CI round trip per batch into a local loop, and it is the only way +to enumerate all batches at once. + +The CI base image (`onnxruntimebuildcache.azurecr.io/.../cpu_x64_almalinux8_gcc14`) requires ACR auth and is +**not pullable**. An equivalent image is easy to build, and reproduces the compiler exactly +(GCC 14.2.1 20250110, Red Hat) — `almalinux:8` + `epel-release` + `gcc-toolset-14` + `python3.11`, with +`cmake ninja packaging numpy setuptools wheel` from pip. Notes: + +- **Python 3.11 is required, not the distro's 3.9** — `tools/ci_build/build_args.py` uses a `match` statement + (3.10+). CI itself uses `/opt/python/cp310-cp310/bin`. +- `--allow_running_as_root` is required inside the container. +- **No X11 dev packages are needed**, because `cmake/external/onnxruntime_external_deps.cmake` sets + `DAWN_USE_X11 OFF` unconditionally. This is what keeps the image minimal. +- Clone into the WSL native filesystem, not `/mnt/d` — building over the 9p mount is crippling. Only + `cmake/external/onnx` and `cmake/external/libprotobuf-mutator` need initialising for a native Linux build. + +To re-check a single file without paying for a whole build, override `CXX_FLAGS` on the generated makefile +rule, which lets you promote warnings back to errors against an existing build tree: + +```bash +FLAGS=$(sed -n 's/^CXX_FLAGS = //p' CMakeFiles/onnxruntime_providers_webgpu.dir/flags.make) +make -f CMakeFiles/onnxruntime_providers_webgpu.dir/build.make \ + CMakeFiles/onnxruntime_providers_webgpu.dir/src//matmul_nbits_mlp.cc.o \ + CXX_FLAGS="$FLAGS -Werror" +``` + +Always run the **negative control** — rebuild the unfixed source through the same harness and confirm it still +errors. Otherwise a harness that silently fails to exercise the warning looks exactly like a successful fix. ## Open items -1. **Fix the Linux leg.** It is currently red on a third `-Werror` batch (`array-bounds`); see - [GCC](#gcc-and-werror). Enumerate the *full* remaining warning list with a local GCC build rather than - discovering one batch per CI round trip. Dispatch a fresh run rather than re-running an old one. +1. **Fix the Linux leg.** The batch-3 `array-bounds` break is fixed at the source, and a local GCC 14 build + confirms it is the *only* ORT-owned warning in the whole build (see + [The complete GCC 14 warning inventory](#the-complete-gcc-14-warning-inventory)), so no further batches are + expected. Not yet confirmed by CI — dispatch a fresh run rather than re-running an old one. 2. ~~**Operator-level validation on a real GPU.**~~ **Done** — see [Results on a real GPU](#results-on-a-real-gpu). No behavioural difference. Remaining sub-item: decide whether to close the 12-test `HardSwish_WebGPU` / `MatMul2BitsWebGpu` coverage gap on the plugin path, or document it as known. From 153df4e3731255a5a2a0600035b7b67e030b3348 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Wed, 26 Aug 2026 18:40:22 -0700 Subject: [PATCH 020/107] Update the work log with the fix verification, push state, and dispatched CI runs Records both halves of the array-bounds verification (single-object recheck with a negative control, plus a full -Werror build), the commit table through ef68884a95, and the three CI runs dispatched on it. Also fixes a broken internal anchor: the heading 'GCC and -Werror' slugs to gcc-and--werror with a double hyphen, so every link to it was dead. Renamed the heading so it slugs unambiguously. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../static_plugin_ep_work_log.md | 34 +++++++++++++++++-- 1 file changed, 31 insertions(+), 3 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md index 990cbedc3458e..22860f7af2c4b 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md @@ -30,7 +30,7 @@ The implementation is complete and committed. CI coverage was added for the new | Shared library plugin build (`--use_webgpu shared_lib`) | Green in CI, no regression from D2/D7 | | CUDA plugin EP (shares `include/onnxruntime/ep/*`) | Green in CI | | Operator-level validation on a real GPU | **Done — no behavioural difference**, see [Testing](#testing) | -| Linux/GCC build (`--use_webgpu static_plugin`) | `array-bounds` fixed at the source; verified locally under GCC 14, **not yet confirmed in CI** — see [GCC](#gcc-and-werror) | +| Linux/GCC build (`--use_webgpu static_plugin`) | `array-bounds` fixed; full local GCC 14 build green under `-Werror`. CI run `33030694040` dispatched, result pending — see [GCC](#gcc-and-warnings-as-errors) | | Minimal build | Not yet exercised — see [Open items](#open-items) | | Emscripten / ORT Web | Not yet exercised — see [Open items](#open-items) | | PR | Not opened yet | @@ -50,6 +50,15 @@ The implementation is complete and committed. CI coverage was added for the new | `49855f6ef7` | Fix GCC maybe-uninitialized errors in the EP adapter attribute accessors | | `d47f00a0a6` | Update dynamic plugin EP test config parsing tests | | `1980e901ec` | Record CI coverage in the static plugin EP registration design doc | +| `754e025ddc` | Add a work log for the static plugin EP branch | +| `a362107df9` | Merge `origin/main` (52 commits behind at the time; merged, not rebased) | +| `35e6f65d52` | Record real-GPU validation results in the work log | +| `a110948b1d` | Record GPU validation and the plugin-path test coverage gap in the design doc | +| `8d87c11972` | Avoid GCC 14 `-Warray-bounds` false positive in `EmitGateActivationExpr` — the batch-3 CI fix | +| `ef68884a95` | Record the complete GCC 14 warning inventory and the local reproduction recipe | + +Pushed through `ef68884a95`. The push from `754e025ddc` was a fast-forward; no history has been rewritten on +this branch. ## What the change consists of @@ -285,7 +294,8 @@ gh api repos/microsoft/onnxruntime/actions/jobs//logs > log.txt When grepping the result, search for `FAILED:` **case-sensitively**. A case-insensitive search for "failed" or a search for `error:` drowns in benign CMake and Dawn feature-probe output such as -`-- Performing Test ... - Failed`. +`-- Performing Test ... - Failed`. The same trap applies to local build logs: `grep -c "error:"` matches Dawn's +own configure line `-- DAWN Werror: OFF`, which looks alarming in a summary count and is nothing. ### Run history @@ -297,8 +307,11 @@ a search for `error:` drowns in benign CMake and Dawn feature-probe output such | 32996643911 | Linux WebGPU CI | Failure — GCC `maybe-uninitialized`, batch 2 | | 32996646394 | CUDA Plugin Windows CI | Success | | 33009901327 | Linux WebGPU CI | Failure — GCC `array-bounds`, batch 3. Internal-EP job on the same run passed. | +| 33030694040 | Linux WebGPU CI | Dispatched on `ef68884a95` — first run carrying the `array-bounds` fix | +| 33030695742 | ONNX Runtime WebGPU Builds | Dispatched on `ef68884a95` — re-run after the merge from `main` | +| 33030697208 | CUDA Plugin Windows CI | Dispatched on `ef68884a95` — re-run after the merge from `main` | -## GCC and `-Werror` +## GCC and warnings as errors This was the only substantial problem the new CI surfaced, and it will bite again, so it is worth understanding rather than pattern-matching. @@ -393,6 +406,21 @@ make -f CMakeFiles/onnxruntime_providers_webgpu.dir/build.make \ Always run the **negative control** — rebuild the unfixed source through the same harness and confirm it still errors. Otherwise a harness that silently fails to exercise the warning looks exactly like a successful fix. +### Verification of the fix + +Both halves were checked, because each one alone is inconclusive: + +1. **Single-object recheck.** With the fix, `matmul_nbits_mlp.cc.o` compiles clean under `-Werror`. Reverting the + file and rebuilding through the identical harness reproduces the exact CI error, which is what proves the + harness was actually exercising the warning. +2. **Full build with `-Werror` on**, i.e. the real CI configuration with `--compile_no_warning_as_error` dropped: + completes to 100%, links `onnxruntime_provider_test`, zero compiler errors. Before trusting this, confirm + `-Werror` really is on the ORT targets — `sed -n 's/^CXX_FLAGS = //p' CMakeFiles/.dir/flags.make` + should show it for `onnxruntime_providers_webgpu`, `onnxruntime_providers` and `onnxruntime_session` — and + confirm the fixed file appears in the build log, so it was recompiled rather than served from a stale object. + +The 334 warnings that remain in that build are all third-party, in targets that do not use `-Werror`. + ## Open items 1. **Fix the Linux leg.** The batch-3 `array-bounds` break is fixed at the source, and a local GCC 14 build From ff3580ce999d2aa5f2a55636455db876276fa5e2 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Wed, 26 Aug 2026 19:22:47 -0700 Subject: [PATCH 021/107] Record the Emscripten/ORT Web build validation for the static plugin EP Builds and links WASM with --use_webgpu static_plugin, and verifies at the symbol level that ORT core's static plugin registration is live rather than compiled out. Also records that Linux WebGPU CI is green after the array-bounds fix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../static_plugin_ep_registration_design.md | 33 ++++++++ .../static_plugin_ep_work_log.md | 83 +++++++++++++++++-- 2 files changed, 107 insertions(+), 9 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md index d62c7456c454c..734ce60a97b7f 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -353,6 +353,39 @@ so a targeted source fix is almost always available and preferable to a blanket When validating such a fix locally, always rebuild the *unfixed* source through the same harness first and confirm it still errors. A harness that silently fails to exercise the warning is indistinguishable from a successful fix. +### Emscripten and ORT Web + +Making the statically linked WebGPU EP reachable through the plugin path is what motivates this design, so the +Emscripten configuration was built before landing the change: `--build_wasm --enable_wasm_simd +--enable_wasm_threads --use_webgpu static_plugin --use_webnn --target onnxruntime_webassembly`. It configures, +compiles and links cleanly, producing `ort-wasm-simd-threaded.asyncify.{mjs,wasm}`. + +`static_plugin` is in fact the only plugin kind Emscripten supports — `cmake/onnxruntime_providers_webgpu.cmake` +raises a `FATAL_ERROR` for `shared_lib` there, since there is no runtime library to load. + +No CI leg covers this yet. The WASM WebGPU jobs pass a bare `--use_webgpu`, which defaults to `static_lib` +(`tools/ci_build/build_args.py` uses `const="static_lib"`), so they still build the internal EP. This change makes +the configuration available to ORT Web; switching ORT Web over to it is separate work. + +A green build is *not* sufficient evidence that the registration is live. `CreateStaticPluginEpLibraries()` in +`onnxruntime/core/session/plugin_ep/ep_static_plugins.cc` is guarded by +`#if defined(USE_WEBGPU) && defined(ORT_WEBGPU_STATIC_PLUGIN)`; if that define does not reach ORT core the function +still compiles, simply returning an empty vector, and the build is just as green while registering nothing. The +check that actually discriminates is at the symbol level — ORT core must hold an *undefined* reference to the +prefixed entry points, resolved by the provider library: + +``` +llvm-nm libonnxruntime_session.a | grep WebGpu_ # U WebGpu_CreateEpFactories / U WebGpu_ReleaseEpFactory +llvm-nm libonnxruntime_providers_webgpu.a | grep WebGpu_ # T WebGpu_CreateEpFactories / T WebGpu_ReleaseEpFactory +``` + +Both were observed on the Emscripten build, which confirms the guard was live, the D3 entry-point prefixing worked, +and the link resolved core's reference against the provider. The binary has not yet been executed in a browser, so +this validates build, link and registration wiring rather than runtime behaviour. + +A practical build note: the WASM build needs `node` on `PATH` — `node_helper.cmake` does a hard `find_program` and +fails configure without it. emsdk bundles a suitable one under `cmake/external/emsdk/node//bin`. + ## Resolution of workstream open questions > Should static factories be registered before environment creation or through environment construction options? diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md index 22860f7af2c4b..14c9018ecbd27 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md @@ -30,9 +30,9 @@ The implementation is complete and committed. CI coverage was added for the new | Shared library plugin build (`--use_webgpu shared_lib`) | Green in CI, no regression from D2/D7 | | CUDA plugin EP (shares `include/onnxruntime/ep/*`) | Green in CI | | Operator-level validation on a real GPU | **Done — no behavioural difference**, see [Testing](#testing) | -| Linux/GCC build (`--use_webgpu static_plugin`) | `array-bounds` fixed; full local GCC 14 build green under `-Werror`. CI run `33030694040` dispatched, result pending — see [GCC](#gcc-and-warnings-as-errors) | +| Linux/GCC build (`--use_webgpu static_plugin`) | **Green in CI** (run `33030694040`) after the `array-bounds` fix — see [GCC](#gcc-and-warnings-as-errors) | | Minimal build | Not yet exercised — see [Open items](#open-items) | -| Emscripten / ORT Web | Not yet exercised — see [Open items](#open-items) | +| Emscripten / ORT Web | **Builds and links**, plugin registration verified in the binary — see [ORT Web](#emscripten-and-ort-web). Not yet run in a browser | | PR | Not opened yet | ## Branch @@ -307,7 +307,7 @@ own configure line `-- DAWN Werror: OFF`, which looks alarming in a summary coun | 32996643911 | Linux WebGPU CI | Failure — GCC `maybe-uninitialized`, batch 2 | | 32996646394 | CUDA Plugin Windows CI | Success | | 33009901327 | Linux WebGPU CI | Failure — GCC `array-bounds`, batch 3. Internal-EP job on the same run passed. | -| 33030694040 | Linux WebGPU CI | Dispatched on `ef68884a95` — first run carrying the `array-bounds` fix | +| 33030694040 | Linux WebGPU CI | **Success** — both jobs. Confirms the `array-bounds` fix; first green GCC run on this branch. | | 33030695742 | ONNX Runtime WebGPU Builds | Dispatched on `ef68884a95` — re-run after the merge from `main` | | 33030697208 | CUDA Plugin Windows CI | Dispatched on `ef68884a95` — re-run after the merge from `main` | @@ -421,12 +421,77 @@ Both halves were checked, because each one alone is inconclusive: The 334 warnings that remain in that build are all third-party, in targets that do not use `-Werror`. +## Emscripten and ORT Web + +This is the motivating use case for the whole change, so it was built locally before opening the PR. **It builds +and links, and the plugin registration is verifiably present in the output.** + +Note what ORT Web's own CI does *not* cover today: the WebGPU WASM jobs in +`.github/workflows/linux-wasm-ci-build-and-test-workflow.yml` pass a bare `--use_webgpu`, and bare `--use_webgpu` +defaults to `static_lib` (`tools/ci_build/build_args.py`, `const="static_lib"`). So every existing WASM job still +builds the **internal** EP. Nothing in CI exercises WASM + `static_plugin`; this branch only makes the +configuration available, it does not switch ORT Web over to it. + +The CMake side was already written with Emscripten in mind — `cmake/onnxruntime_providers_webgpu.cmake` rejects +`shared_lib` there with `"WebGPU EP shared library build is not supported on Emscripten. Please use +'--use_webgpu static_plugin'."` + +### Building it + +Same container as the GCC 14 reproduction (see [above](#reproducing-the-ci-compiler-locally)), plus: + +- `git submodule update --init --recursive cmake/external/emsdk` — not needed for a native Linux build, required + here. `build.py` installs and activates the toolchain itself. +- **`node` must be on `PATH`.** `cmake/external/onnxruntime_webassembly.cmake` includes `node_helper.cmake`, + which does a hard `find_program(NODE_EXECUTABLE ...)` and fails configure with + `Could not find NODE_EXECUTABLE using the following names: node.exe, node`. CI gets node from + `actions/setup-node`; the AlmaLinux image has none. emsdk bundles one — point `PATH` at + `cmake/external/emsdk/node//bin`. + +``` +docker run --rm -v /home/edch/ort:/src -v /home/edch/ortwasm:/build -w /src ort-gcc14:local \ + bash -lc "export PATH=/src/cmake/external/emsdk/node/22.16.0_64bit/bin:\$PATH && \ + python3.11 tools/ci_build/build.py --build_dir /build --config Release --skip_submodule_sync --parallel \ + --allow_running_as_root --build_wasm --enable_wasm_simd --enable_wasm_threads \ + --enable_wasm_api_exception_catching --use_webgpu static_plugin --use_webnn \ + --target onnxruntime_webassembly --skip_tests" +``` + +Result: configure clean, zero compile failures, links `ort-wasm-simd-threaded.asyncify.mjs` + +`.wasm` (~32 MB). Roughly 35 minutes including the emsdk toolchain download and sysroot generation. + +### Verifying the plugin path actually linked + +A successful build alone proves little here, because the registration is behind +`#if defined(USE_WEBGPU) && defined(ORT_WEBGPU_STATIC_PLUGIN)` in +`onnxruntime/core/session/plugin_ep/ep_static_plugins.cc`. If that define failed to reach ORT core, the file +would still compile — `CreateStaticPluginEpLibraries()` would just return an empty vector, and the build would +be just as green while registering nothing. + +The decisive check is that ORT core holds an **undefined** reference to the prefixed entry points, using the +emsdk LLVM tools: + +``` +NM=cmake/external/emsdk/upstream/bin/llvm-nm +$NM /Release/libonnxruntime_session.a | grep WebGpu_ # expect: U WebGpu_CreateEpFactories, U WebGpu_ReleaseEpFactory +$NM /Release/libonnxruntime_providers_webgpu.a | grep WebGpu_ # expect: T WebGpu_CreateEpFactories, T WebGpu_ReleaseEpFactory +``` + +Observed exactly that: `U` in core, `T` in the provider, and the link resolved them. That closes the loop — +the `#if` was live, the entry point prefixing worked, and `EpLibraryStaticPlugin` is in the binary. + +### Still open + +The binary has not been **run**. `--skip_tests` was used, and exercising it needs a browser with WebGPU, which +the container does not have. So this validates build, link and registration wiring on Emscripten, not runtime +behaviour. + ## Open items -1. **Fix the Linux leg.** The batch-3 `array-bounds` break is fixed at the source, and a local GCC 14 build - confirms it is the *only* ORT-owned warning in the whole build (see - [The complete GCC 14 warning inventory](#the-complete-gcc-14-warning-inventory)), so no further batches are - expected. Not yet confirmed by CI — dispatch a fresh run rather than re-running an old one. +1. ~~**Fix the Linux leg.**~~ **Done.** The batch-3 `array-bounds` break is fixed at the source, and a local + GCC 14 build confirmed it was the *only* ORT-owned warning in the whole build (see + [The complete GCC 14 warning inventory](#the-complete-gcc-14-warning-inventory)). CI run `33030694040` + is green on both jobs. 2. ~~**Operator-level validation on a real GPU.**~~ **Done** — see [Results on a real GPU](#results-on-a-real-gpu). No behavioural difference. Remaining sub-item: decide whether to close the 12-test `HardSwish_WebGPU` / `MatMul2BitsWebGpu` coverage gap on the plugin path, or document it as known. @@ -435,8 +500,8 @@ The 334 warnings that remain in that build are all third-party, in targets that registration call in `ort_env.cc` is wrapped in `#if !defined(ORT_MINIMAL_BUILD)`. The guard has been inspected and looks correct but has not been exercised. Also unknown: whether any shipped ORT Web configuration uses an extended-minimal build, which would exclude the plugin EP infrastructure entirely. -4. **Emscripten / ORT Web.** This is the motivating use case for the whole change and has not been built. - `web.yml` exists and supports `workflow_dispatch`. +4. ~~**Emscripten / ORT Web.**~~ **Builds and links**, with the registration verified in the binary — see + [Emscripten and ORT Web](#emscripten-and-ort-web). Not yet run in a browser. 5. **Follow-up cleanup** listed at the end of the design doc, including removing the WebGPU special cases that only exist while it is still an "internal" EP. 6. **Open the PR.** From d83e02807b3a41de88c0eb6cc205df0015ab6428 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 27 Aug 2026 16:47:17 -0700 Subject: [PATCH 022/107] Record the ORT Web migration prerequisites in the static plugin EP design doc Building the WebGPU EP into the WASM binary as a static plugin is only half of moving ORT Web onto the plugin path. Document the two remaining gaps, neither of which the linker can catch: ORT Web selects the EP by name and that name resolves only to the internal EP, and no WebGPU OrtEpDevice exists under Emscripten because the default device discovery reports a CPU only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../static_plugin_ep_registration_design.md | 50 ++++++++++++++++++- 1 file changed, 48 insertions(+), 2 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md index 734ce60a97b7f..259e5343d6807 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -365,7 +365,8 @@ raises a `FATAL_ERROR` for `shared_lib` there, since there is no runtime library No CI leg covers this yet. The WASM WebGPU jobs pass a bare `--use_webgpu`, which defaults to `static_lib` (`tools/ci_build/build_args.py` uses `const="static_lib"`), so they still build the internal EP. This change makes -the configuration available to ORT Web; switching ORT Web over to it is separate work. +the configuration available to ORT Web; switching ORT Web over to it is separate work, and has prerequisites that +this change does not address. See [ORT Web migration](#ort-web-migration). A green build is *not* sufficient evidence that the registration is live. `CreateStaticPluginEpLibraries()` in `onnxruntime/core/session/plugin_ep/ep_static_plugins.cc` is guarded by @@ -386,7 +387,52 @@ this validates build, link and registration wiring rather than runtime behaviour A practical build note: the WASM build needs `node` on `PATH` — `node_helper.cmake` does a hard `find_program` and fails configure without it. emsdk bundles a suitable one under `cmake/external/emsdk/node//bin`. -## Resolution of workstream open questions +## ORT Web migration + +The intended end state is that the WebGPU EP is always a plugin EP and `--use_webgpu static_lib` goes away +entirely, ORT Web included. Building the EP into the WASM binary as a static plugin, which this change enables, is +only the first half of that. The second half is *selection*, and two gaps stand between the current state and a +working ORT Web on the plugin path. Neither is visible to the linker, so the symbol-level evidence above does not +speak to either. + +**ORT Web selects the EP by name, and that name resolves only to the internal EP.** +`js/web/lib/wasm/session-options.ts` maps the `'webgpu'` backend to `epName = 'WebGPU'`, which reaches +`OrtAppendExecutionProvider` in `onnxruntime/wasm/api.cc` and from there +`OrtApis::SessionOptionsAppendExecutionProvider`. That API is a closed table of hardcoded EP names in +`onnxruntime/core/session/provider_registration.cc`, and its `EpID::WebGPU` case is compiled out under +`ORT_USE_EP_API_ADAPTERS`. On a `static_plugin` build the EP therefore registers at `OrtEnv` creation and is then +unselectable from JS: the call returns "WebGPU execution provider is not supported in this build". This is by +design rather than an oversight — the name-based API is the internal-EP mechanism, and plugin EPs are selected +through `RegisterExecutionProviderLibrary` plus `OrtEpDevice`-based selection — but it does mean ORT Web needs new +plumbing rather than a recompile. + +**No WebGPU `OrtEpDevice` exists under Emscripten.** Emscripten is neither `WIN32`, `LINUX` nor `APPLE`, so +`cmake/onnxruntime_common.cmake` selects `core/platform/device_discovery_default.cc`, which discovers only a CPU +device. `Factory::GetSupportedDevicesImpl` in `onnxruntime/core/providers/webgpu/ep/factory.cc` creates an +`OrtEpDevice` only for devices of type `OrtHardwareDeviceType_GPU`. The one fallback, the virtual GPU device in the +same function, is opt-in through the `allow_virtual_devices` environment configuration entry and is deliberately +registered without allocator info, because it exists to back a device-free compile-only session. So even after +selection plumbing is added, `GetEpDevices` would offer nothing for JS to select. + +A workable order: + +1. Make WebGPU visible as a device under Emscripten, either through an Emscripten device discovery implementation + that reports a GPU or by having the factory synthesize a GPU `OrtEpDevice` there. The latter fits better: WebGPU + availability in a browser is a JS fact rather than an OS enumeration fact, and ORT Web already supplies the + actual `GPUDevice` through EP options, which makes the `OrtEpDevice` effectively a handle. It must not reuse the + virtual-device path, which is allocator-less by design and cannot back a real session. +2. Expose plugin EP selection in the WASM C API. `g_env` is already available in `api.cc`, so `GetEpDevices` and + `SessionOptionsAppendExecutionProvider_V2` are both reachable. A narrow helper that appends by EP name and + resolves the device internally avoids marshalling arrays of device pointers across the JS boundary. The new + entry point needs to be exported from `cmake/onnxruntime_webassembly.cmake`. +3. Switch `js/web` to the new call. The `BUILD_DEFS.DISABLE_WEBGPU` branch that falls back to the JS EP is + unaffected. Confirm that the EP options ORT Web passes today — `deviceId`, `webgpuInstance`, `webgpuDevice`, + `preferredLayout` and the buffer cache modes — still reach the provider, since `Factory::CreateEpImpl` obtains + them from the session config options. +4. Add a CI leg. No pipeline builds WASM with `static_plugin` today, so without one this path stays validated at + the symbol level only. + + > Should static factories be registered before environment creation or through environment construction options? From 7db781fc38f55ccd010e8a316bbc81122d3f5816 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 27 Aug 2026 16:53:45 -0700 Subject: [PATCH 023/107] Reject a minimal build combined with the WebGPU static plugin EP A minimal build excludes core/session/plugin_ep from the session library, so Environment::CreateAndRegisterStaticPluginEps is compiled out, but the WebGPU provider is still built and linked. The combination produced a binary holding the EP that never registered it, with no diagnostic at configure, build or run time. Fail configure instead. This is a temporary guard. Letting minimal builds use a static plugin EP is worthwhile and needs the registration path to be available there first. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- cmake/CMakeLists.txt | 9 +++++++++ .../static_plugin_ep_registration_design.md | 11 +++++++++++ .../static_plugin_ep_work_log.md | 12 +++++++----- 3 files changed, 27 insertions(+), 5 deletions(-) diff --git a/cmake/CMakeLists.txt b/cmake/CMakeLists.txt index d0c9d7aedd704..851780cb39d6d 100644 --- a/cmake/CMakeLists.txt +++ b/cmake/CMakeLists.txt @@ -450,6 +450,15 @@ if(onnxruntime_WEBGPU_STATIC_PLUGIN) if(NOT onnxruntime_USE_EP_API_ADAPTERS) message(FATAL_ERROR "onnxruntime_WEBGPU_STATIC_PLUGIN=ON requires onnxruntime_USE_EP_API_ADAPTERS=ON.") endif() + # A minimal build excludes core/session/plugin_ep, so the registration that a static plugin EP depends on is not + # compiled in. Without this check the EP would be linked into the binary but never registered, with no diagnostic + # at configure, build or run time. Supporting the combination is possible but requires making the plugin EP + # registration available to minimal builds first. + if(onnxruntime_MINIMAL_BUILD) + message(FATAL_ERROR "onnxruntime_WEBGPU_STATIC_PLUGIN=ON is not supported in a minimal build: the plugin EP " + "registration in core/session/plugin_ep is excluded, so the EP would be linked in but " + "never registered.") + endif() add_compile_definitions(ORT_WEBGPU_STATIC_PLUGIN=1) endif() diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md index 259e5343d6807..c2f5038e1a47a 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -106,6 +106,17 @@ other two no, so only the linkage and test sites move to the derived option. The `build.py` gains `--use_webgpu static_plugin`. +`onnxruntime_WEBGPU_STATIC_PLUGIN` validates its prerequisites at configure time and fails with `FATAL_ERROR` if +they are not met: it requires `onnxruntime_USE_WEBGPU` and `onnxruntime_USE_EP_API_ADAPTERS`, and it rejects a +minimal build. The last one is a temporary guard rather than a permanent restriction. A minimal build excludes +`core/session/plugin_ep` from `onnxruntime_session_srcs`, and `Environment::CreateAndRegisterStaticPluginEps` is +compiled out with it, but `cmake/onnxruntime_providers_webgpu.cmake` still builds and links the provider. The +combination would therefore produce a binary containing the WebGPU EP that never registers it, with nothing +reporting the problem at configure, build or run time. Allowing minimal builds to use a static plugin EP is +worthwhile and needs the registration path to be available there first; the exclusion is currently justified by +provider-bridge dependencies, and only two of the files under `plugin_ep` are provider-bridge, so the subset may be +separable. Until then a loud configure error is preferable to a silently EP-less binary. + The CUDA plugin is unaffected: it is gated by `onnxruntime_BUILD_CUDA_EP_AS_PLUGIN` and sets `ORT_USE_EP_API_ADAPTERS` as a private target compile definition, never referencing the CMake option. CUDA's split between role (`BUILD_CUDA_EP_AS_PLUGIN`) and mechanism (`ORT_USE_EP_API_ADAPTERS`) is the in-tree template for diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md index 14c9018ecbd27..04309e6f32feb 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md @@ -495,11 +495,13 @@ behaviour. 2. ~~**Operator-level validation on a real GPU.**~~ **Done** — see [Results on a real GPU](#results-on-a-real-gpu). No behavioural difference. Remaining sub-item: decide whether to close the 12-test `HardSwish_WebGPU` / `MatMul2BitsWebGpu` coverage gap on the plugin path, or document it as known. -3. **Minimal build.** `linux_minimal_build.yml` has not been dispatched. It is the interesting one because - `cmake/onnxruntime_session.cmake` excludes `plugin_ep/*.*` from minimal builds, which is why the - registration call in `ort_env.cc` is wrapped in `#if !defined(ORT_MINIMAL_BUILD)`. The guard has been - inspected and looks correct but has not been exercised. Also unknown: whether any shipped ORT Web - configuration uses an extended-minimal build, which would exclude the plugin EP infrastructure entirely. +3. ~~**Minimal build.**~~ **Resolved.** Exactly one pipeline combines an extended-minimal build with WebGPU — + `webgpu_minimal_build_edge_build_x64_RelWithDebInfo` in `.github/workflows/windows_webgpu.yml`, which passes a + bare `--use_webgpu` (internal EP) and is green, so the `#if !defined(ORT_MINIMAL_BUILD)` guard in `ort_env.cc` + is already exercised in CI. No ORT Web or wasm job uses a minimal build at all. The remaining hole — minimal + plus `static_plugin` linking the EP in without registering it — is now a configure-time `FATAL_ERROR` in + `cmake/CMakeLists.txt`, verified to fire on that combination and not on a non-minimal `static_plugin` + configure. Lifting the restriction properly is follow-up work, recorded in D4 of the design doc. 4. ~~**Emscripten / ORT Web.**~~ **Builds and links**, with the registration verified in the binary — see [Emscripten and ORT Web](#emscripten-and-ort-web). Not yet run in a browser. 5. **Follow-up cleanup** listed at the end of the design doc, including removing the WebGPU special cases that From 014b8de190c6e97776019f2096279349d7e754f0 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 27 Aug 2026 19:25:18 -0700 Subject: [PATCH 024/107] Record ORT Web plugin-EP prototype results in design doc Ran the full js/web WebGPU operator suite (2152 tests) in Edge against a WASM --use_webgpu static_plugin binary. All pass. Update the ORT Web migration section to record what the prototype established for each step: Emscripten OrtEpDevice synthesis, the three-place WASM export requirement, EP option prefix equivalence between the name-based and V2 paths, the canonical EP name change needed in session-options.ts, and the test-runner-cli --webgpu-ep forwarding gap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../static_plugin_ep_registration_design.md | 43 ++++++++++++++++++- 1 file changed, 41 insertions(+), 2 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md index c2f5038e1a47a..025b059be747f 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -392,8 +392,9 @@ llvm-nm libonnxruntime_providers_webgpu.a | grep WebGpu_ # T WebGpu_CreateEpFa ``` Both were observed on the Emscripten build, which confirms the guard was live, the D3 entry-point prefixing worked, -and the link resolved core's reference against the provider. The binary has not yet been executed in a browser, so -this validates build, link and registration wiring rather than runtime behaviour. +and the link resolved core's reference against the provider. That check validates build, link and registration +wiring; runtime behaviour was established separately by the ORT Web prototype described under +[ORT Web migration](#ort-web-migration). A practical build note: the WASM build needs `node` on `PATH` — `node_helper.cmake` does a hard `find_program` and fails configure without it. emsdk bundles a suitable one under `cmake/external/emsdk/node//bin`. @@ -406,6 +407,11 @@ only the first half of that. The second half is *selection*, and two gaps stand working ORT Web on the plugin path. Neither is visible to the linker, so the symbol-level evidence above does not speak to either. +Both gaps have since been closed by a throwaway prototype, which ran the full `js/web` WebGPU operator suite in +Edge against a WASM `static_plugin` binary: 2152 tests, all passing. The design below is therefore validated end +to end rather than merely plausible; what remains is productionization, not discovery. The prototype's findings are +recorded inline in the steps that follow. + **ORT Web selects the EP by name, and that name resolves only to the internal EP.** `js/web/lib/wasm/session-options.ts` maps the `'webgpu'` backend to `epName = 'WebGPU'`, which reaches `OrtAppendExecutionProvider` in `onnxruntime/wasm/api.cc` and from there @@ -432,17 +438,50 @@ A workable order: availability in a browser is a JS fact rather than an OS enumeration fact, and ORT Web already supplies the actual `GPUDevice` through EP options, which makes the `OrtEpDevice` effectively a handle. It must not reuse the virtual-device path, which is allocator-less by design and cannot back a real session. + + *Prototype result:* factory synthesis works. When the real-GPU loop yields no devices, the factory creates a + GPU `OrtHardwareDevice` with null metadata — null metadata is what keeps it out of the virtual-device + classification, which is keyed off `kOrtHardwareDevice_MetadataKey_IsVirtual` — and registers both the default + and the read-only allocator info. That is sufficient for session creation and shared-allocator setup; no + Emscripten device discovery implementation is needed. + 2. Expose plugin EP selection in the WASM C API. `g_env` is already available in `api.cc`, so `GetEpDevices` and `SessionOptionsAppendExecutionProvider_V2` are both reachable. A narrow helper that appends by EP name and resolves the device internally avoids marshalling arrays of device pointers across the JS boundary. The new entry point needs to be exported from `cmake/onnxruntime_webassembly.cmake`. + + *Prototype result:* the name-resolving helper is the right shape. Note that exporting it takes three edits, not + one: `EMSCRIPTEN_KEEPALIVE` in `api.h`, the `JSPI_EXPORTS` list in `cmake/onnxruntime_webassembly.cmake` for + JSPI builds, and the `wrapAsyncAPIs` list in `onnxruntime/wasm/pre-async.js` for Asyncify builds. Omitting the + last one is the easy mistake, and Asyncify is the default variant. Reporting matters too: on no match the helper + should raise a status naming the EPs that *are* registered, which is what makes a misconfiguration diagnosable + from JS through `checkLastError`. + 3. Switch `js/web` to the new call. The `BUILD_DEFS.DISABLE_WEBGPU` branch that falls back to the JS EP is unaffected. Confirm that the EP options ORT Web passes today — `deviceId`, `webgpuInstance`, `webgpuDevice`, `preferredLayout` and the buffer cache modes — still reach the provider, since `Factory::CreateEpImpl` obtains them from the session config options. + + *Prototype result:* the options do reach the provider unchanged, and the reason is worth recording. Both paths + derive their config prefix from `OrtSessionOptions::GetProviderOptionPrefix` keyed on the EP name, so V2 lands + the options under exactly the same `ep.webgpuexecutionprovider.*` keys the name-based path used. One JS change + is required though: `session-options.ts` passes the short name `'WebGPU'`, whereas `OrtEpDevice` selection + matches the canonical `'WebGpuExecutionProvider'` returned by `EpDevice_EpName`. + + The productionization question the prototype leaves open is *conditionality*. The prototype switches to V2 + unconditionally whenever WebGPU is enabled, which is not shippable: the same JS is bundled against both + `static_lib` and `static_plugin` binaries, and nothing at TypeScript build time distinguishes them. This wants + either a build define threaded through from the WASM build or a runtime capability probe. + 4. Add a CI leg. No pipeline builds WASM with `static_plugin` today, so without one this path stays validated at the symbol level only. + *Prototype note:* the `js/web` test runner cannot currently exercise this configuration end to end, because + `script/test-runner-cli.ts` spawns `script/build` with only `--bundle-mode` and does not forward `--webgpu-ep`. + Setting `npm_config_webgpu_ep` in the environment works as a stopgap, since `script/build.ts` reads it, but a CI + leg should forward the flag properly. Note also that without `--webgpu-ep` the `webgpu` backend silently routes + through JSEP instead, so a leg that omits it would pass while testing nothing relevant. + > Should static factories be registered before environment creation or through environment construction options? From 018b24ae932877cb6856c39851152d608d0d89a4 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Fri, 28 Aug 2026 10:18:04 -0700 Subject: [PATCH 025/107] Add Emscripten device discovery reporting a GPU when navigator.gpu exists Emscripten previously fell through to device_discovery_default.cc, which reports only a CPU device, so no OrtEpDevice could ever be created for WebGPU in a wasm build. Add core/platform/emscripten/device_discovery.cc, selected by a new Emscripten arm in onnxruntime_common.cmake. It reports the CPU device plus a GPU OrtHardwareDevice when a synchronous EM_ASM_INT check finds navigator.gpu. No factory change is needed: the existing loop in Factory::GetSupportedDevicesImpl filters on OrtHardwareDeviceType_GPU, so the discovered device flows through and picks up both allocator infos. vendor_id/device_id are left at 0 because a browser will not identify the GPU: requestAdapter() is async and unavailable at OrtEnv creation time, and GPUAdapterInfo masks vendor details anyway. The device is deliberately not tagged is_virtual, since it is backed by real hardware. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- cmake/onnxruntime_common.cmake | 3 + .../static_plugin_ep_registration_design.md | 69 +++++++++++++------ .../platform/emscripten/device_discovery.cc | 59 ++++++++++++++++ 3 files changed, 110 insertions(+), 21 deletions(-) create mode 100644 onnxruntime/core/platform/emscripten/device_discovery.cc diff --git a/cmake/onnxruntime_common.cmake b/cmake/onnxruntime_common.cmake index 659699c370f1f..6df5f00c74cda 100644 --- a/cmake/onnxruntime_common.cmake +++ b/cmake/onnxruntime_common.cmake @@ -104,6 +104,9 @@ elseif (LINUX) elseif (APPLE) list(APPEND onnxruntime_common_src_patterns "${ONNXRUNTIME_ROOT}/core/platform/apple/device_discovery.cc") +elseif (CMAKE_SYSTEM_NAME STREQUAL "Emscripten") + list(APPEND onnxruntime_common_src_patterns + "${ONNXRUNTIME_ROOT}/core/platform/emscripten/device_discovery.cc") else() list(APPEND onnxruntime_common_src_patterns "${ONNXRUNTIME_ROOT}/core/platform/device_discovery_default.cc") diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md index 025b059be747f..963ce955db2dc 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -423,9 +423,9 @@ design rather than an oversight — the name-based API is the internal-EP mechan through `RegisterExecutionProviderLibrary` plus `OrtEpDevice`-based selection — but it does mean ORT Web needs new plumbing rather than a recompile. -**No WebGPU `OrtEpDevice` exists under Emscripten.** Emscripten is neither `WIN32`, `LINUX` nor `APPLE`, so -`cmake/onnxruntime_common.cmake` selects `core/platform/device_discovery_default.cc`, which discovers only a CPU -device. `Factory::GetSupportedDevicesImpl` in `onnxruntime/core/providers/webgpu/ep/factory.cc` creates an +**No WebGPU `OrtEpDevice` exists under Emscripten.** Emscripten was neither `WIN32`, `LINUX` nor `APPLE`, so +`cmake/onnxruntime_common.cmake` fell through to `core/platform/device_discovery_default.cc`, which discovers only +a CPU device. `Factory::GetSupportedDevicesImpl` in `onnxruntime/core/providers/webgpu/ep/factory.cc` creates an `OrtEpDevice` only for devices of type `OrtHardwareDeviceType_GPU`. The one fallback, the virtual GPU device in the same function, is opt-in through the `allow_virtual_devices` environment configuration entry and is deliberately registered without allocator info, because it exists to back a device-free compile-only session. So even after @@ -433,17 +433,37 @@ selection plumbing is added, `GetEpDevices` would offer nothing for JS to select A workable order: -1. Make WebGPU visible as a device under Emscripten, either through an Emscripten device discovery implementation - that reports a GPU or by having the factory synthesize a GPU `OrtEpDevice` there. The latter fits better: WebGPU - availability in a browser is a JS fact rather than an OS enumeration fact, and ORT Web already supplies the - actual `GPUDevice` through EP options, which makes the `OrtEpDevice` effectively a handle. It must not reuse the - virtual-device path, which is allocator-less by design and cannot back a real session. - - *Prototype result:* factory synthesis works. When the real-GPU loop yields no devices, the factory creates a - GPU `OrtHardwareDevice` with null metadata — null metadata is what keeps it out of the virtual-device - classification, which is keyed off `kOrtHardwareDevice_MetadataKey_IsVirtual` — and registers both the default - and the read-only allocator info. That is sufficient for session creation and shared-allocator setup; no - Emscripten device discovery implementation is needed. +1. Make WebGPU visible as a device under Emscripten by adding an Emscripten device discovery implementation. + `cmake/onnxruntime_common.cmake` gains an `Emscripten` arm selecting a new + `core/platform/emscripten/device_discovery.cc`, which reports the CPU device via `GetCpuDeviceFromCPUIDInfo()` + plus a GPU `OrtHardwareDevice` when a synchronous `EM_ASM_INT` check finds `navigator.gpu`. No change to + `Factory::GetSupportedDevicesImpl` is needed: its existing loop filters on `OrtHardwareDeviceType_GPU`, so a + discovered GPU flows through and picks up both allocator infos unmodified. + + The prototype first tried the other option — having the factory synthesize the device itself through + `CreateHardwareDevice` — and it worked, but it is the wrong layer. `GetSupportedDevices` documents its `devices` + argument as "the `OrtHardwareDevice` instances that are available", so that array is the source of truth, and + `CreateHardwareDevice` is documented for devices with no hardware behind them ("e.g., virtual"). A browser GPU + is real hardware that ORT's discovery layer merely failed to enumerate, so the producer is the place to fix it. + Leaving it in the factory would also oblige every future Emscripten plugin EP to repeat the same workaround. + + A capability check is a legitimate form of discovery here. `core/platform/apple/device_discovery.cc` sets the + precedent: it enumerates nothing, it asserts that one GPU exists on Apple Silicon and hardcodes the vendor id. + `navigator.gpu` is a stronger signal than that — an actual runtime capability probe, and the same one ORT Web's + `initEp` uses to decide whether WebGPU is usable. `EM_ASM` from ORT core is likewise established practice + (`core/graph/model.cc`, `core/framework/external_data_loader.cc`), not just an EP-layer habit. + + The honest limitation is that no adapter information is available. `_OrtInit` creates the `OrtEnv` and therefore + triggers discovery, and it runs before `navigator.gpu.requestAdapter()` in ORT Web's `initEp`, so `vendor_id` + and `device_id` are left at 0. That is acceptable: Apple reports no device id either, and browsers mask + `GPUAdapterInfo.vendor` for fingerprinting resistance regardless. The device is deliberately *not* tagged with + `kOrtHardwareDevice_MetadataKey_IsVirtual`, which is what keeps it distinct from the virtual device — that path + is allocator-less by design and cannot back a real session. + + One accepted side effect: `ep_library_internal.cc` creates an internal `WebGpuEpFactory` in a + `USE_WEBGPU && !ORT_USE_EP_API_ADAPTERS` build, so today's `static_lib` ORT Web build starts producing an + `OrtEpDevice` where it previously produced none. That build selects by name and is slated for removal, so the + change is benign. 2. Expose plugin EP selection in the WASM C API. `g_env` is already available in `api.cc`, so `GetEpDevices` and `SessionOptionsAppendExecutionProvider_V2` are both reachable. A narrow helper that appends by EP name and @@ -468,13 +488,20 @@ A workable order: is required though: `session-options.ts` passes the short name `'WebGPU'`, whereas `OrtEpDevice` selection matches the canonical `'WebGpuExecutionProvider'` returned by `EpDevice_EpName`. - The productionization question the prototype leaves open is *conditionality*. The prototype switches to V2 - unconditionally whenever WebGPU is enabled, which is not shippable: the same JS is bundled against both - `static_lib` and `static_plugin` binaries, and nothing at TypeScript build time distinguishes them. This wants - either a build define threaded through from the WASM build or a runtime capability probe. - -4. Add a CI leg. No pipeline builds WASM with `static_plugin` today, so without one this path stays validated at - the symbol level only. + The productionization question this raises is *conditionality*: is it safe to switch to V2 unconditionally + whenever WebGPU is enabled? It is, provided the WASM build flips at the same time. `BUILD_DEFS.DISABLE_WEBGPU` + is defined in `js/web/script/build.ts` as exactly `!USE_WEBGPU_EP`, so the bundle is already specialized per + WASM flavour by `--webgpu-ep` — the JSEP bundle takes the other branch entirely and is unaffected. Flipping the + WebGPU WASM builds to `static_plugin` (step 4) therefore makes `!DISABLE_WEBGPU` imply `static_plugin` by + construction, and no additional build define or runtime capability probe is needed. What is *not* safe is + landing the JS change without the build change, since the two must move together. + +4. Flip the ORT Web WASM builds to `static_plugin` and add a CI leg. The two builds are + `Build (simd + threads + WebGPU experimental)` and its JSPI sibling in + `.github/workflows/linux-wasm-ci-build-and-test-workflow.yml`, both of which pass a bare `--use_webgpu` and so + get `static_lib` from `build_args.py`'s `const="static_lib"`. Neither uses `--minimal_build`, so the + minimal-build `FATAL_ERROR` guard does not stand in the way of this migration. No pipeline builds WASM with + `static_plugin` today, so without a leg this path stays validated at the symbol level only. *Prototype note:* the `js/web` test runner cannot currently exercise this configuration end to end, because `script/test-runner-cli.ts` spawns `script/build` with only `--bundle-mode` and does not forward `--webgpu-ep`. diff --git a/onnxruntime/core/platform/emscripten/device_discovery.cc b/onnxruntime/core/platform/emscripten/device_discovery.cc new file mode 100644 index 0000000000000..0b4b544a573ee --- /dev/null +++ b/onnxruntime/core/platform/emscripten/device_discovery.cc @@ -0,0 +1,59 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +#include "core/platform/device_discovery.h" + +#include + +namespace onnxruntime { + +namespace { + +// Returns true if the JS environment exposes a WebGPU entry point. +// +// This is a capability check rather than a hardware enumeration, because a browser does not offer anything +// better. A GPUAdapter is only obtainable through the asynchronous navigator.gpu.requestAdapter(), which cannot +// be called from here, and even once obtained GPUAdapterInfo masks vendor details for fingerprinting +// resistance. The presence of navigator.gpu is therefore the strongest signal available at discovery time. It is +// also the same signal ORT Web itself uses to decide whether WebGPU is usable (see initEp() in +// js/web/lib/wasm/wasm-core-impl.ts). +// +// navigator is defined in both window and worker scopes, so this works on either. It is undefined, or present +// without a gpu property, in a non-browser host such as Node.js, in which case no GPU is reported. +bool IsWebGpuAvailable() { + // clang-format off + // The body below is JavaScript, not C++; clang-format would otherwise rewrite `!==` as `!= =`. + return EM_ASM_INT({ + return (typeof navigator !== 'undefined' && navigator.gpu) ? 1 : 0; + }) != 0; + // clang-format on +} + +} // namespace + +std::unordered_set DeviceDiscovery::DiscoverDevicesForPlatform() { + std::unordered_set devices{}; + + devices.emplace(GetCpuDeviceFromCPUIDInfo()); + + // Report a GPU when the host exposes WebGPU. There is no OS-level device enumeration under Emscripten and a + // browser will not identify the GPU it is running on, but what matters for EP selection is that a GPU exists + // and is reachable, which is exactly what navigator.gpu establishes. + // + // vendor_id and device_id are left at 0 because they are unknown, not because the device is synthetic: this + // entry describes real hardware, so it is deliberately not marked with kOrtHardwareDevice_MetadataKey_IsVirtual + // and is a valid device to back an inference session. + if (IsWebGpuAvailable()) { + OrtHardwareDevice gpu_device{}; + gpu_device.type = OrtHardwareDeviceType_GPU; + gpu_device.vendor_id = 0; + gpu_device.device_id = 0; + gpu_device.metadata.Add("Description", "WebGPU"); + + devices.emplace(std::move(gpu_device)); + } + + return devices; +} + +} // namespace onnxruntime From 93b8f1419917d709802fb7eddd4f65cf4ba701b4 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Fri, 28 Aug 2026 14:53:30 -0700 Subject: [PATCH 026/107] Don't use exceptions for control flow in EP API OpKernelInfo adapter GetAttr/GetAttrs were implemented as a try around the throwing Ort:: C++ wrappers, converting Ort::Exception into a Status. GetAttrOrDefault relies on that non-OK Status to substitute the default, which makes a missing optional attribute depend on catching an exception. Emscripten builds using --enable_wasm_api_exception_catching compile only the C API boundary translation units with exception catching enabled, so the catch never matched and the exception escaped GetAttrOrDefault. This showed up as 949 suite0 failures on a reduced-size WebGPU static_plugin wasm build. Call the non-throwing OrtApi::KernelInfoGetAttribute* functions directly and convert the returned OrtStatus* instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../static_plugin_ep_registration_design.md | 102 +++++++++++++++++ .../onnxruntime/ep/adapter/op_kernel_info.h | 106 +++++++++++++++--- 2 files changed, 194 insertions(+), 14 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md index 963ce955db2dc..5702caee1fc30 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -509,6 +509,108 @@ A workable order: leg should forward the flag properly. Note also that without `--webgpu-ep` the `webgpu` backend silently routes through JSEP instead, so a leg that omits it would pass while testing nothing relevant. +### Interaction with the ORT Web reduced-size build options + +The ORT Web WASM builds are size-sensitive and pass a set of size-reduction flags that the local prototype builds +did not. `linux-wasm-ci-build-and-test-workflow.yml` applies `--disable_ml_ops --disable_generation_ops +--disable_types string float4 float8 optional sparsetensor --include_ops_by_config +onnxruntime/wasm/reduced_types.config --enable_reduced_operator_type_support`, and `web.yml` adds `--disable_rtti` +for the release job. Since the migration changes how the WebGPU EP is compiled and linked, each of these was +checked against the plugin path. + +**None of them need adapting.** The findings: + +- *Compile definitions reach the WebGPU target in every mode.* `onnxruntime_providers_webgpu` is created through + `onnxruntime_add_static_library` / `onnxruntime_add_shared_library_module`, both of which call + `onnxruntime_configure_target` → `onnxruntime_set_compile_flags`. That function is where every `DISABLE_*` and + `REDUCED_OPS_BUILD` definition is applied, so the `static_lib`, `static_plugin` and shared-library builds all + receive an identical set. There is no propagation gap introduced by the plugin boundary. + +- *Operator and type reduction never applied to WebGPU, before or after.* + `op_registration_utils.get_kernel_registration_files()` hardcodes the CPU registration files (plus CUDA when + requested); WebGPU is not in the list, so `--include_ops_by_config` does not rewrite WebGPU kernel + registrations. `onnxruntime/wasm/reduced_types.config` is `!no_ops_specified_means_all_ops_are_required` — it + performs global *type* reduction only, with no op exclusion — and that mechanism works through the + `op_kernel_type_control` macros, which neither `core/providers/webgpu` nor `contrib_ops/webgpu` uses. Its saving + is entirely CPU-EP-side and is unchanged by the plugin switch. This is a pre-existing gap, not a regression. + +- *The type and ML/generation op flags are inert for WebGPU.* No source under `core/providers/webgpu` or + `contrib_ops/webgpu` references `DISABLE_ML_OPS`, `DISABLE_GENERATION_OPS`, `DISABLE_SPARSE_TENSORS`, + `DISABLE_OPTIONAL_TYPE`, `DISABLE_FLOAT8_TYPES`, `DISABLE_FLOAT4_TYPES` or `DISABLE_STRING_TYPE`. + +- *`--disable_rtti` is safe on the plugin path.* Neither `include/onnxruntime/ep` (the EP API adapters) nor + `onnxruntime/core/session/plugin_ep` uses `dynamic_cast` or `typeid`. + +One consequence is worth recording for the *shared library* plugin EP, which is out of scope here but shares this +code. `onnxruntime_c_api.h` contains no `DISABLE_*` guards, so the `OrtApi` struct layout is invariant under these +flags and the ABI is stable. `onnxruntime_cxx_api.h` does guard some `Ort::Value` members (the sparse tensor +methods) — an EP DLL built with a different `--disable_types` than its host would therefore see a different C++ +header surface, but since those wrappers are inline over stable function pointers this is a source-compatibility +concern rather than an ABI one. For `static_plugin` everything is compiled in one tree with one set of defines, so +the question does not arise. + +**What is *not* covered by any existing flag** is the plugin machinery itself. Relative to `static_lib`, the +`static_plugin` build adds `core/providers/webgpu/ep/*` (excluded from the source list in the `static_lib` branch +of `onnxruntime_providers_webgpu.cmake`), inlines the header-only EP API adapters into every WebGPU translation +unit, and reaches the `OrtGraph` / `OrtNode` C API graph views through `OrtEp::GetCapability`. Note that +`core/session/plugin_ep` is *not* part of that delta: `onnxruntime_session.cmake` only excludes it for +`onnxruntime_MINIMAL_BUILD`, so today's WASM builds already compile it in. + +That also means the only existing lever that trims plugin-EP machinery is `--minimal_build`, which +`static_plugin` currently rejects outright. If measurement shows the size delta matters, lifting that restriction +(already listed as a follow-up) is the mechanism to reach parity — not a new size-reduction flag. + +#### Measured size comparison + +Both modes were then built locally with the exact CI flag set (`Release`, SIMD + threads, asyncify, WebNN on, +`--disable_rtti`, `--enable_wasm_api_exception_catching`, and the full reduced-size set above) and the artifacts +compared: + +| Artifact | `static_lib` | `static_plugin` | Delta | +| --- | ---: | ---: | ---: | +| `ort-wasm-simd-threaded.asyncify.wasm` | 26,243,969 B | 26,190,991 B | **−52,978 B (−0.20%)** | +| `ort-wasm-simd-threaded.asyncify.mjs` | 53,218 B | 53,218 B | 0 | + +**There is no size regression** — `static_plugin` is marginally smaller. Both configurations compiled cleanly with +every reduced-size flag, which empirically confirms the analysis above. The plugin build plausibly comes out ahead +because it drops the internal `IExecutionProvider` / `KernelRegistry` glue and the built-in `WebGpuEpFactory` in +`ep_library_internal.cc` in favour of the header-only adapters, but that attribution is not separately measured. +For reference, the same `static_plugin` configuration without any reduced-size flags is 40.9 MB, so those flags are +worth ~15 MB and remain essential regardless of EP mode. + +#### `--enable_wasm_api_exception_catching` and the EP API adapters + +The one option that *did* interact with the plugin path is `--enable_wasm_api_exception_catching`. It is not a +size-reduction flag in the same sense as the others, but it is part of the same CI flag set and it exposed a real +bug. + +`onnxruntime_webassembly.cmake` compiles only `wasm/api.cc` and `core/session/onnxruntime_c_api.cc` with +`-sDISABLE_EXCEPTION_CATCHING=0`; every other translation unit keeps Emscripten's default, where `catch` clauses +are compiled so that they never match. Exceptions still propagate — they are simply only catchable at the C API +boundary. + +The EP API adapters in `include/onnxruntime/ep/adapter/` are header-only and are therefore inlined into WebGPU +translation units that have catching disabled. `OpKernelInfo::GetAttr()` / `GetAttrs()` were implemented as a +`try` around the throwing `Ort::` C++ wrappers, converting `Ort::Exception` into a `Status`. That makes a *missing +optional attribute* — ordinary control flow for `GetAttrOrDefault()` — depend on catching an exception. With +catching disabled the `catch` never ran, so the exception escaped `GetAttrOrDefault()` entirely and surfaced from +session creation as e.g. `ERROR_CODE: 6, ERROR_MESSAGE: No attribute with name:'extrapolation_value'is defined.` + +This reproduced as 949 `suite0` failures on the reduced-size `static_plugin` build while the same build passed on +the CPU backend, the reduced-size `static_lib` build passed, and the full-op `static_plugin` build passed — the +failure required plugin + WebGPU + API-only exception catching together. + +The fix is to not use exceptions for control flow in the adapters: `GetAttr()` / `GetAttrs()` now call the +non-throwing `OrtApi::KernelInfoGetAttribute*` functions directly and convert the returned `OrtStatus*` into a +`Status`. **General rule: header-only EP API adapter code must not rely on catching exceptions**, because a plugin +EP may be compiled with exception catching disabled. The remaining `catch` blocks in `include/onnxruntime/ep` +(`common.h`, `get_capability_utils.h`, `adapter/kernel_registry.h`) sit on genuinely exceptional callback +boundaries rather than on control-flow paths; when catching is disabled those exceptions still reach the outer C +API boundary and are reported, only with a less specific message. + +With that fix the reduced-size `static_plugin` build passes `suite0` on the WebGPU backend in full: 2152 tests, +all passing, matching the `static_lib` control run on the identical flag set. + > Should static factories be registered before environment creation or through environment construction options? diff --git a/include/onnxruntime/ep/adapter/op_kernel_info.h b/include/onnxruntime/ep/adapter/op_kernel_info.h index 2e03caa60cdc9..aa6c784c0a9c1 100644 --- a/include/onnxruntime/ep/adapter/op_kernel_info.h +++ b/include/onnxruntime/ep/adapter/op_kernel_info.h @@ -9,7 +9,11 @@ #include #include +#include +#include +#include +#include "core/common/common.h" #include "core/common/inlined_containers.h" #include "core/common/narrow.h" #include "core/common/status.h" @@ -154,8 +158,8 @@ struct OpKernelInfo { template [[nodiscard]] T GetAttrOrDefault(const std::string& name, const T& default_value) const { - // Value-initialize. These accessors are fully inline here, so the compiler can see the failure and - // exception paths of GetAttr() and warns about a possibly uninitialized read without it. + // Value-initialize. These accessors are fully inline here, so the compiler can see the failure + // paths of GetAttr() and warns about a possibly uninitialized read without it. T tmp{}; return GetAttr(name, &tmp).IsOK() ? tmp : default_value; } @@ -172,21 +176,11 @@ struct OpKernelInfo { } template Status GetAttr(const std::string& name, T* value) const { - try { - *value = info_.GetAttribute(name.c_str()); - return Status::OK(); - } catch (const Ort::Exception& ex) { - return Status(onnxruntime::common::ONNXRUNTIME, ex.GetOrtErrorCode(), ex.what()); - } + return GetAttrImpl(cache_->kernel_info_, name.c_str(), value); } template Status GetAttrs(const std::string& name, std::vector& values) const { - try { - values = info_.GetAttributes(name.c_str()); - return Status::OK(); - } catch (const Ort::Exception& ex) { - return Status(onnxruntime::common::ONNXRUNTIME, ex.GetOrtErrorCode(), ex.what()); - } + return GetAttrsImpl(cache_->kernel_info_, name.c_str(), values); } Status GetAttrs(const std::string& name, TensorShapeVector& out) const { @@ -212,6 +206,90 @@ struct OpKernelInfo { } private: + // A missing optional attribute is normal control flow for GetAttrOrDefault()/GetAttrsOrDefault(), so these + // accessors must not depend on catching the exception thrown by the Ort:: C++ wrappers: a plugin EP may be + // compiled with C++ exception catching disabled. The ORT Web build is one such case -- it builds with + // `-sDISABLE_EXCEPTION_CATCHING` everywhere except the C API boundary, so a `catch` in this inlined header + // never matches and the exception escapes all the way out of session creation. Use the non-throwing C API + // directly instead. + static Status ToStatus(OrtStatus* ort_status) { + if (ort_status == nullptr) { + return Status::OK(); + } + const Ort::Status status{ort_status}; // takes ownership + return Status(onnxruntime::common::ONNXRUNTIME, status.GetErrorCode(), status.GetErrorMessage()); + } + + static Status GetAttrImpl(const OrtKernelInfo* info, const char* name, float* out) { + return ToStatus(Ort::GetApi().KernelInfoGetAttribute_float(info, name, out)); + } + + static Status GetAttrImpl(const OrtKernelInfo* info, const char* name, int64_t* out) { + return ToStatus(Ort::GetApi().KernelInfoGetAttribute_int64(info, name, out)); + } + + static Status GetAttrImpl(const OrtKernelInfo* info, const char* name, std::string* out) { + size_t size = 0; + // Feed nullptr for the data buffer to query the true size of the string attribute. + ORT_RETURN_IF_ERROR(ToStatus(Ort::GetApi().KernelInfoGetAttribute_string(info, name, nullptr, &size))); + + std::string value; + value.resize(size); + ORT_RETURN_IF_ERROR(ToStatus(Ort::GetApi().KernelInfoGetAttribute_string(info, name, value.data(), &size))); + value.resize(size - 1); // remove the terminating character '\0' + *out = std::move(value); + return Status::OK(); + } + + static Status GetAttrsImpl(const OrtKernelInfo* info, const char* name, std::vector& out) { + size_t size = 0; + ORT_RETURN_IF_ERROR(ToStatus(Ort::GetApi().KernelInfoGetAttributeArray_float(info, name, nullptr, &size))); + + std::vector values(size); + ORT_RETURN_IF_ERROR(ToStatus(Ort::GetApi().KernelInfoGetAttributeArray_float(info, name, values.data(), &size))); + out.swap(values); + return Status::OK(); + } + + static Status GetAttrsImpl(const OrtKernelInfo* info, const char* name, std::vector& out) { + size_t size = 0; + ORT_RETURN_IF_ERROR(ToStatus(Ort::GetApi().KernelInfoGetAttributeArray_int64(info, name, nullptr, &size))); + + std::vector values(size); + ORT_RETURN_IF_ERROR(ToStatus(Ort::GetApi().KernelInfoGetAttributeArray_int64(info, name, values.data(), &size))); + out.swap(values); + return Status::OK(); + } + + static Status GetAttrsImpl(const OrtKernelInfo* info, const char* name, std::vector& out) { + Ort::AllocatorWithDefaultOptions allocator; + size_t size = 0; + ORT_RETURN_IF_ERROR( + ToStatus(Ort::GetApi().KernelInfoGetAttributeArray_string(info, name, allocator, nullptr, &size))); + if (size == 0) { + out.clear(); + return Status::OK(); + } + + char** raw_values = nullptr; + ORT_RETURN_IF_ERROR( + ToStatus(Ort::GetApi().KernelInfoGetAttributeArray_string(info, name, allocator, &raw_values, &size))); + + std::vector values; + values.reserve(size); + for (size_t i = 0; i < size; ++i) { + if (raw_values[i] != nullptr) { + values.emplace_back(raw_values[i]); + allocator.Free(raw_values[i]); + } else { + values.emplace_back(); + } + } + allocator.Free(raw_values); + out.swap(values); + return Status::OK(); + } + const Ort::ConstKernelInfo info_; std::shared_ptr cache_; }; From e856885e88d1fd0ff53cc15dff2e341364e94dfa Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Fri, 28 Aug 2026 15:11:50 -0700 Subject: [PATCH 027/107] Record WebGPU static_plugin vs static_lib perf comparison in design doc Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../static_plugin_ep_registration_design.md | 42 +++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md index 5702caee1fc30..ae024af0a7007 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -611,6 +611,48 @@ API boundary and are reported, only with a less specific message. With that fix the reduced-size `static_plugin` build passes `suite0` on the WebGPU backend in full: 2152 tests, all passing, matching the `static_lib` control run on the identical flag set. +#### Measured performance comparison + +The same two binaries were compared on inference performance in Edge, using the test runner's `--perf` mode +(`test-runner-cli model -b=webgpu -e=edge -P=`) against two synthetic models chosen to isolate the two +places the plugin path could plausibly cost something: + +- **dispatch-bound** — 300 chained tiny elementwise ops (`Mul`/`Add`/`Sub`) on a `[1, 1024]` tensor. Almost no GPU + work per node, so the run time is dominated by per-node host-side work, which is exactly what now goes through + the C API adapters. +- **compute-bound** — 16 chained `[512, 512]` `MatMul`s. Host-side per-node cost is negligible against the GPU + work. + +Three runs per model per build; the table reports the median of the three per-run P50 values. + +| Metric | `static_lib` | `static_plugin` | Delta | +| --- | ---: | ---: | ---: | +| dispatch-bound, per-run P50 | 10.90 ms | 11.80 ms | +0.9 ms (+8%) | +| compute-bound, per-run P50 | 7.00 ms | 7.30 ms | +0.3 ms (+4%, within noise) | +| session init, dispatch model | 1136 ms | 1172 ms | +36 ms (+3%) | +| session init, compute model | 1250 ms | 1290 ms | +40 ms (+3%) | +| `suite0` net test time | 44.95 / 45.28 s | 45.71 / 44.93 s | none measurable | + +Reading of these numbers: + +- **Execution is not measurably slower in any realistic case.** The compute-bound difference is inside run-to-run + variance, and `suite0`'s net test time — 2152 models, two samples per build — is indistinguishable between the + two builds. +- **A dispatch-bound graph pays roughly 8%.** That is the cost of routing per-node kernel work through the C API + adapters instead of a direct in-process virtual call. It only shows up when nodes do essentially no GPU work, + which is the worst case by construction rather than a representative one. Amortized over ~300 nodes it is on the + order of 3 µs per node. +- **Session creation costs about 3%.** This covers plugin EP registration, `GetCapability` and kernel creation + through the C API. Note the absolute figure is dominated by fetching and parsing the model, so the true relative + cost of the plugin machinery within session creation is higher than 3% — but it is tens of milliseconds once, + not per inference. + +A caution on methodology: the *total* wall time the test runner prints for `suite0` is not a usable metric. It +varied between 3 min 15 s and 4 min 04 s for the **same** `static_lib` binary across two runs, and `static_plugin` +produced both 4 min 13 s and 3 min 16 s. Only the second figure it prints — net test time, excluding the +per-model `before all` session-creation hooks — is stable enough to compare, and by that measure the two builds +are equal. An early single-sample comparison of the total figure suggested a 29% regression that does not exist. + > Should static factories be registered before environment creation or through environment construction options? From 6abb02df3e6912a5e470fa1224a68c0cee1bb5a1 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Fri, 28 Aug 2026 17:43:59 -0700 Subject: [PATCH 028/107] Select WebGPU as a plugin EP from ORT Web Add OrtAppendExecutionProviderV2 to the wasm API, which selects an EP by device rather than by built-in EP name, and route the 'webgpu' backend through it so ORT Web uses the statically linked plugin EP. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- cmake/onnxruntime_webassembly.cmake | 2 +- js/web/lib/wasm/session-options.ts | 15 +++++++++++++-- js/web/lib/wasm/wasm-types.ts | 7 +++++++ onnxruntime/wasm/api.cc | 30 +++++++++++++++++++++++++++++ onnxruntime/wasm/api.h | 16 +++++++++++++++ onnxruntime/wasm/pre-async.js | 2 ++ 6 files changed, 69 insertions(+), 3 deletions(-) diff --git a/cmake/onnxruntime_webassembly.cmake b/cmake/onnxruntime_webassembly.cmake index fa93113c76160..9d909e1f9a7af 100644 --- a/cmake/onnxruntime_webassembly.cmake +++ b/cmake/onnxruntime_webassembly.cmake @@ -274,7 +274,7 @@ else() if (onnxruntime_ENABLE_WEBASSEMBLY_JSPI) target_link_options(onnxruntime_webassembly PRIVATE "SHELL:-s JSPI=1" - "SHELL:-s JSPI_EXPORTS=[OrtAppendExecutionProvider,OrtCreateSession,OrtRun,OrtRunWithBinding,OrtBindInput]" + "SHELL:-s JSPI_EXPORTS=[OrtAppendExecutionProvider,OrtAppendExecutionProviderV2,OrtCreateSession,OrtRun,OrtRunWithBinding,OrtBindInput]" ) else() # NOTE: "-s ASYNCIFY=1" is required for JSEP to work with WebGPU diff --git a/js/web/lib/wasm/session-options.ts b/js/web/lib/wasm/session-options.ts index 7f7bb35ae58e0..827d87c6bbf3b 100644 --- a/js/web/lib/wasm/session-options.ts +++ b/js/web/lib/wasm/session-options.ts @@ -79,6 +79,10 @@ const setExecutionProviders = async ( for (const ep of executionProviders) { let epName = typeof ep === 'string' ? ep : ep.name; const epOptions: Array<[number, number]> = []; + // True when the EP is selected as a plugin EP (by OrtEpDevice) rather than by ORT's built-in EP name table. + // The built-in name table has no WebGPU entry in an ORT_USE_EP_API_ADAPTERS build, which is what the wasm + // binary paired with this bundle is built as (see BUILD_DEFS.DISABLE_WEBGPU below). + let selectAsPluginEp = false; // check EP name switch (epName) { @@ -99,7 +103,10 @@ const setExecutionProviders = async ( break; case 'webgpu': if (!BUILD_DEFS.DISABLE_WEBGPU) { - epName = 'WebGPU'; + // The plugin EP is registered under its canonical name (OrtEpFactory::GetName), not the short 'WebGPU' + // alias that ORT's built-in EP name table used. + epName = 'WebGpuExecutionProvider'; + selectAsPluginEp = true; let customDevice: GPUDevice | undefined; if (typeof ep !== 'string') { @@ -197,8 +204,12 @@ const setExecutionProviders = async ( getInstance().setValue(valuesOffset + i * getInstance().PTR_SIZE, epOptions[i][1], '*'); } } + const appendFn = selectAsPluginEp + ? getInstance()._OrtAppendExecutionProviderV2 + : getInstance()._OrtAppendExecutionProvider; if ( - (await getInstance()._OrtAppendExecutionProvider( + (await appendFn.call( + getInstance(), sessionOptionsHandle, epNameDataOffset, keysOffset, diff --git a/js/web/lib/wasm/wasm-types.ts b/js/web/lib/wasm/wasm-types.ts index 82d6808196a5f..6e6d1676d503e 100644 --- a/js/web/lib/wasm/wasm-types.ts +++ b/js/web/lib/wasm/wasm-types.ts @@ -413,6 +413,13 @@ export interface OrtInferenceAPIs { providerOptionsValues: number, numKeys: number, ): Promise; + _OrtAppendExecutionProviderV2( + sessionOptionsHandle: number, + name: number, + providerOptionsKeys: number, + providerOptionsValues: number, + numKeys: number, + ): Promise; _OrtAddFreeDimensionOverride(sessionOptionsHandle: number, name: number, dim: number): number; _OrtAddSessionConfigEntry(sessionOptionsHandle: number, configKey: number, configValue: number): number; _OrtReleaseSessionOptions(sessionOptionsHandle: number): number; diff --git a/onnxruntime/wasm/api.cc b/onnxruntime/wasm/api.cc index 0494c7471f2ac..a8c1c56a00fa6 100644 --- a/onnxruntime/wasm/api.cc +++ b/onnxruntime/wasm/api.cc @@ -16,6 +16,7 @@ WGPUDevice GetDevice(int); } // namespace onnxruntime #endif +#include #include #include #include @@ -180,6 +181,35 @@ int OrtAppendExecutionProvider(ort_session_options_handle_t session_options, return CHECK_STATUS(SessionOptionsAppendExecutionProvider, session_options, name, provider_options_keys, provider_options_values, num_keys); } +int OrtAppendExecutionProviderV2(ort_session_options_handle_t session_options, + const char* name, + const char* const* provider_options_keys, + const char* const* provider_options_values, + size_t num_keys) { + const OrtEpDevice* const* ep_devices = nullptr; + size_t num_ep_devices = 0; + RETURN_ERROR_CODE_IF_ERROR(GetEpDevices, g_env, &ep_devices, &num_ep_devices); + + for (size_t i = 0; i < num_ep_devices; ++i) { + const char* ep_name = Ort::GetApi().EpDevice_EpName(ep_devices[i]); + if (ep_name != nullptr && std::strcmp(ep_name, name) == 0) { + return CHECK_STATUS(SessionOptionsAppendExecutionProvider_V2, session_options, g_env, + &ep_devices[i], static_cast(1), + provider_options_keys, provider_options_values, num_keys); + } + } + + std::ostringstream message; + message << "No execution provider device is registered for '" << name << "'. Registered devices:"; + if (num_ep_devices == 0) { + message << " (none)"; + } + for (size_t i = 0; i < num_ep_devices; ++i) { + message << " '" << Ort::GetApi().EpDevice_EpName(ep_devices[i]) << "'"; + } + return CheckStatus(Ort::GetApi().CreateStatus(ORT_NOT_FOUND, message.str().c_str())); +} + int OrtAddFreeDimensionOverride(ort_session_options_handle_t session_options, const char* dim_param_name, int dim_value) { diff --git a/onnxruntime/wasm/api.h b/onnxruntime/wasm/api.h index 0f75b8c3defed..b07931f72c6da 100644 --- a/onnxruntime/wasm/api.h +++ b/onnxruntime/wasm/api.h @@ -94,6 +94,22 @@ int EMSCRIPTEN_KEEPALIVE OrtAppendExecutionProvider(ort_session_options_handle_t const char* const* provider_options_values, size_t num_keys); +/** + * append an execution provider for a session, selecting it as a plugin EP by name. + * + * Unlike OrtAppendExecutionProvider, which resolves the name against ORT's built-in EP table, this looks the + * name up among the OrtEpDevice instances registered with the environment and appends the matching one through + * SessionOptionsAppendExecutionProvider_V2. It is how a statically linked plugin EP is selected. + * + * @param name the EP name as reported by the EP factory, e.g. "WebGpuExecutionProvider" + * @returns ORT error code. If not zero, call OrtGetLastError() to get detailed error message. + */ +int EMSCRIPTEN_KEEPALIVE OrtAppendExecutionProviderV2(ort_session_options_handle_t session_options, + const char* name, + const char* const* provider_options_keys, + const char* const* provider_options_values, + size_t num_keys); + /** * add a free dimension override for one dimension of a session's input. */ diff --git a/onnxruntime/wasm/pre-async.js b/onnxruntime/wasm/pre-async.js index 1f8f17535e7d4..a19e3c3be1fdf 100644 --- a/onnxruntime/wasm/pre-async.js +++ b/onnxruntime/wasm/pre-async.js @@ -16,6 +16,7 @@ let initAsyncImpl = () => { // // Currently in ASYNCIFY build, we only use this for the following functions: // - OrtAppendExecutionProvider() + // - OrtAppendExecutionProviderV2() // - OrtCreateSession() // - OrtRun() // - OrtRunWithBinding() @@ -49,6 +50,7 @@ let initAsyncImpl = () => { wrapAsyncAPIs([ "_OrtAppendExecutionProvider", + "_OrtAppendExecutionProviderV2", "_OrtCreateSession", "_OrtRun", "_OrtRunWithBinding", From 6a3e4bdb1aad26d506e369bf3e59d59e7dcdd990 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Fri, 28 Aug 2026 18:23:51 -0700 Subject: [PATCH 029/107] Note upstream allow_software_adapter fallback in Emscripten device discovery rationale --- .../static_plugin_ep_registration_design.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md index ae024af0a7007..c83b0171c123d 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -460,6 +460,14 @@ A workable order: `kOrtHardwareDevice_MetadataKey_IsVirtual`, which is what keeps it distinct from the virtual device — that path is allocator-less by design and cannot back a real session. + A later change on `main` added a second fallback to `GetSupportedDevicesImpl` alongside the virtual device: when + no GPU device is discovered, `allow_software_adapter` (opt-in via the `kAllowSoftwareAdapterEnvironmentVariable` + environment variable) advertises WebGPU against the *CPU* `OrtHardwareDevice`. It does not change the analysis + above. Both upstream fallbacks work around a device the discovery layer failed to enumerate; they are opt-in + precisely because misreporting a device is not safe to do by default. Under Emscripten the device is genuinely + there and cheaply detectable, so fixing discovery keeps the default path correct and needs no opt-in. The + primary matching loop that our discovered GPU device flows through is unchanged by that work. + One accepted side effect: `ep_library_internal.cc` creates an internal `WebGpuEpFactory` in a `USE_WEBGPU && !ORT_USE_EP_API_ADAPTERS` build, so today's `static_lib` ORT Web build starts producing an `OrtEpDevice` where it previously produced none. That build selects by name and is slated for removal, so the From b876d290cdc51506dbea35d46d3f92ff10e84024 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Mon, 31 Aug 2026 18:29:50 -0700 Subject: [PATCH 030/107] Defer zero-initialize submission in the WebGPU plugin EP shared allocator BufferManager::Create clears a buffer recycled from the cache, and GpuBufferAllocator's should_submit_zero_initialize predicate decides whether that clear is submitted before Create returns. During Session::Run it must not be, so the clear batches with the run's own dispatches. The plugin factory's shared allocator, which backs intermediate tensors, hardcoded the predicate to true and forced a queue submit for every intermediate buffer allocated mid-run. A 16-node MatMul benchmark ran 36% slower than the statically linked EP as a result. The shared allocator is created before any EP instance exists, so it had no EP to query. Move the run-active state to WebGpuContext, which owns the command encoder that batching depends on and is reachable from the allocator, and have WebGpuExecutionProvider update it and delegate IsRunActive() to it so all three allocator sites share one source of truth. Track it as a count rather than a flag. A WebGpuContext is reference counted per context id and is shared by every session using that device. WebGpuExecutionProvider reports ConcurrentRunSupported() as false, so InferenceSession serializes runs, but it does so with a per-session lock that does not serialize sessions sharing a context. With a flag, one session ending its run would report the context as idle while another session's run was still in flight, and that run would go back to submitting eagerly. The count is signed so an unbalanced decrement fails toward "no run active", which only costs an eager submit, rather than wrapping to a value that looks permanently active. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../core/providers/webgpu/ep/factory.cc | 20 +++++------ .../core/providers/webgpu/webgpu_context.h | 34 +++++++++++++++++++ .../webgpu/webgpu_execution_provider.cc | 11 ++++-- .../webgpu/webgpu_execution_provider.h | 11 ++++-- .../providers/webgpu/webgpu_context_test.cc | 30 ++++++++++++++++ 5 files changed, 91 insertions(+), 15 deletions(-) diff --git a/onnxruntime/core/providers/webgpu/ep/factory.cc b/onnxruntime/core/providers/webgpu/ep/factory.cc index 3ecf8c2b36cc1..a4bf61f751ac0 100644 --- a/onnxruntime/core/providers/webgpu/ep/factory.cc +++ b/onnxruntime/core/providers/webgpu/ep/factory.cc @@ -248,16 +248,16 @@ OrtStatus* ORT_API_CALL Factory::CreateAllocatorImpl( "Unsupported memory info for shared allocator."); } - *allocator = new onnxruntime::ep::adapter::Allocator(memory_info, - [](const OrtMemoryInfo&) -> AllocatorPtr { - return std::make_shared( - []() -> const webgpu::BufferManager& { - return WebGpuContextFactory::DefaultContext() - .BufferManager(); - }, - false, - []() { return true; }); - }); + // Defer the zero-initialize submission while a run is in flight so that the clear batches with the run's own + // dispatches. This allocator is not bound to an EP instance, so it reads the run state from the context. + auto create_allocator = [](const OrtMemoryInfo&) -> AllocatorPtr { + return std::make_shared( + []() -> const webgpu::BufferManager& { return WebGpuContextFactory::DefaultContext().BufferManager(); }, + false, + []() { return !WebGpuContextFactory::DefaultContext().IsRunActive(); }); + }; + + *allocator = new onnxruntime::ep::adapter::Allocator(memory_info, create_allocator); return nullptr; EXCEPTION_TO_RETURNED_STATUS_END } diff --git a/onnxruntime/core/providers/webgpu/webgpu_context.h b/onnxruntime/core/providers/webgpu/webgpu_context.h index 1790c85e7214a..93efa3a80eee2 100644 --- a/onnxruntime/core/providers/webgpu/webgpu_context.h +++ b/onnxruntime/core/providers/webgpu/webgpu_context.h @@ -3,6 +3,9 @@ #pragma once +#include +#include +#include #include #include #include @@ -305,6 +308,32 @@ class WebGpuContext final { void CollectProfilingData(); void EndProfiling(TimePoint, profiling::Events& events); + // Whether a Session::Run is currently in flight on this context. + // + // Buffer zero-initialization consults this to decide whether the clear must be submitted before + // returning: during a run the clear is encoded into the current command encoder and rides along with + // the run's own submission, which preserves dispatch batching. Outside a run there is no upcoming + // submission to piggyback on, so the clear is submitted immediately. + // + // This lives on the context (rather than only on WebGpuExecutionProvider) because the context owns the + // command encoder that batching depends on, and because allocators that are not bound to an EP instance + // must be able to observe it. + // + // A context is shared by every session that uses the same device. WebGpuExecutionProvider reports + // ConcurrentRunSupported() == false, so InferenceSession serializes runs, but that lock is per-session + // and does not serialize sessions sharing a context. This is therefore a count rather than a flag: + // with a flag, one session ending its run would report the context as idle while another session's + // run is still in flight. + bool IsRunActive() const { return active_run_count_.load() > 0; } + + // Must be called in pairs. WebGpuExecutionProvider::OnRunStart increments only on the path that + // returns success, because InferenceSession::Run arranges for OnRunEnd only when OnRunStart succeeded. + void IncrementActiveRunCount() { active_run_count_.fetch_add(1); } + void DecrementActiveRunCount() { + [[maybe_unused]] const auto previous_count = active_run_count_.fetch_sub(1); + assert(previous_count > 0); + } + // // Push error scope. // @@ -414,6 +443,11 @@ class WebGpuContext final { uint32_t num_pending_dispatches_ = 0; uint32_t max_num_pending_dispatches_ = 16; + // Number of Session::Run calls currently in flight on this context, across every session sharing it. + // Signed so that an unbalanced decrement fails toward "no run active" (an eager submit, which is + // always safe) instead of wrapping to a huge value that would look like a run is permanently active. + std::atomic active_run_count_{0}; + // Owns the active dispatch window and the unique pending builds referenced within that window. std::vector deferred_dispatches_; diff --git a/onnxruntime/core/providers/webgpu/webgpu_execution_provider.cc b/onnxruntime/core/providers/webgpu/webgpu_execution_provider.cc index 81cc974906cf3..0659ac70a29cc 100644 --- a/onnxruntime/core/providers/webgpu/webgpu_execution_provider.cc +++ b/onnxruntime/core/providers/webgpu/webgpu_execution_provider.cc @@ -854,12 +854,19 @@ Status WebGpuExecutionProvider::OnRunStart(const onnxruntime::RunOptions& run_op } } - run_active_.store(true); + // Incremented last so that it is only incremented on the path that returns success: InferenceSession::Run + // calls OnRunEnd only for providers whose OnRunStart succeeded, so an early return above must not have + // incremented the count. + context_.IncrementActiveRunCount(); return Status::OK(); } +bool WebGpuExecutionProvider::IsRunActive() const { + return context_.IsRunActive(); +} + Status WebGpuExecutionProvider::OnRunEnd(bool /* sync_stream */, const onnxruntime::RunOptions& run_options) { - run_active_.store(false); + context_.DecrementActiveRunCount(); // When capturing, flushing creates the replay-ready CapturedCommandInfo entries before // CaptureEnd() detaches their external storage. diff --git a/onnxruntime/core/providers/webgpu/webgpu_execution_provider.h b/onnxruntime/core/providers/webgpu/webgpu_execution_provider.h index fcf9db36c3b65..e1cf340af51cb 100644 --- a/onnxruntime/core/providers/webgpu/webgpu_execution_provider.h +++ b/onnxruntime/core/providers/webgpu/webgpu_execution_provider.h @@ -4,7 +4,6 @@ #pragma once -#include #include #include #include @@ -97,7 +96,14 @@ class WebGpuExecutionProvider : public IExecutionProvider { Status OnRunStart(const onnxruntime::RunOptions& run_options) override; Status OnRunEnd(bool sync_stream, const onnxruntime::RunOptions& run_options) override; - bool IsRunActive() const { return run_active_.load(); } + // Run state lives on the WebGpuContext because it owns the command encoder whose dispatch batching + // depends on it, and because allocators not bound to an EP instance (the plugin EP's shared allocator, + // created by the factory before any EP exists) must observe the same state. + // + // This reports context-wide activity: it is true while any session sharing this context has a run in + // flight, not just this one. That is what the batching decision depends on, since the command encoder + // being batched into is shared. + bool IsRunActive() const; // WebGPU EP reuses the Device ID as the key to get the WebGpuContext instance. int GetDeviceId() const override { return context_id_; } @@ -138,7 +144,6 @@ class WebGpuExecutionProvider : public IExecutionProvider { std::vector force_cpu_node_names_; bool enable_graph_capture_ = false; bool graph_buffer_mgr_active_ = false; - std::atomic run_active_{false}; bool enable_int64_ = false; uint32_t multi_rotary_cache_concat_offset_ = 0; uint32_t kv_cache_quantization_bits_ = 0; diff --git a/onnxruntime/test/providers/webgpu/webgpu_context_test.cc b/onnxruntime/test/providers/webgpu/webgpu_context_test.cc index 70d2af300e816..23da0d1d0bb48 100644 --- a/onnxruntime/test/providers/webgpu/webgpu_context_test.cc +++ b/onnxruntime/test/providers/webgpu/webgpu_context_test.cc @@ -222,6 +222,36 @@ TEST(WebGpuContextTest, WebGpuExecutionProviderTracksRunActivity) { EXPECT_FALSE(webgpu_ep->IsRunActive()); } +// Sessions that use the same device share one WebGpuContext, so run activity is tracked as a count. One +// session ending its run must not report the context as idle while another session's run is in flight. +TEST(WebGpuContextTest, RunActivityTracksEveryProviderSharingAContext) { + ConfigOptions options; + auto first_ep = WebGpuProviderFactoryCreator::Create(options)->CreateProvider(); + auto second_ep = WebGpuProviderFactoryCreator::Create(options)->CreateProvider(); + ASSERT_NE(first_ep, nullptr); + ASSERT_NE(second_ep, nullptr); + auto* first_webgpu_ep = static_cast(first_ep.get()); + auto* second_webgpu_ep = static_cast(second_ep.get()); + ASSERT_EQ(first_webgpu_ep->GetDeviceId(), second_webgpu_ep->GetDeviceId()); + RunOptions run_options; + + EXPECT_FALSE(first_webgpu_ep->IsRunActive()); + + ASSERT_STATUS_OK(first_webgpu_ep->OnRunStart(run_options)); + ASSERT_STATUS_OK(second_webgpu_ep->OnRunStart(run_options)); + // Both providers observe the shared context's activity, not just their own. + EXPECT_TRUE(first_webgpu_ep->IsRunActive()); + EXPECT_TRUE(second_webgpu_ep->IsRunActive()); + + ASSERT_STATUS_OK(first_webgpu_ep->OnRunEnd(false, run_options)); + EXPECT_TRUE(first_webgpu_ep->IsRunActive()); + EXPECT_TRUE(second_webgpu_ep->IsRunActive()); + + ASSERT_STATUS_OK(second_webgpu_ep->OnRunEnd(false, run_options)); + EXPECT_FALSE(first_webgpu_ep->IsRunActive()); + EXPECT_FALSE(second_webgpu_ep->IsRunActive()); +} + TEST(WebGpuContextTest, EnablesLazyClearResourceOnFirstUse) { #if defined(__wasm__) || defined(USE_EXTERNAL_DAWN) GTEST_SKIP() << "Dawn native toggle inspection is unavailable."; From 2831d7da1774e0cb3642022bee7bdf7ccd514adc Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Mon, 31 Aug 2026 18:30:02 -0700 Subject: [PATCH 031/107] Update measured size and perf comparison with post-merge interleaved results Re-measures both builds after merging main, using interleaved A/B sampling. Sequential sampling aliased machine drift onto the build under test and understated a real regression by nearly an order of magnitude; documents that, and the shared allocator defect it exposed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../static_plugin_ep_registration_design.md | 88 +++++++++++++------ 1 file changed, 63 insertions(+), 25 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md index c83b0171c123d..136b41eafc8b5 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -576,8 +576,11 @@ compared: | Artifact | `static_lib` | `static_plugin` | Delta | | --- | ---: | ---: | ---: | -| `ort-wasm-simd-threaded.asyncify.wasm` | 26,243,969 B | 26,190,991 B | **−52,978 B (−0.20%)** | -| `ort-wasm-simd-threaded.asyncify.mjs` | 53,218 B | 53,218 B | 0 | +| `ort-wasm-simd-threaded.asyncify.wasm` | 26,550,879 B | 26,488,911 B | **−61,968 B (−0.23%)** | +| `ort-wasm-simd-threaded.asyncify.mjs` | 53,249 B | 53,249 B | 0 | + +(Both binaries rebuilt after merging `main`. An earlier pre-merge pair measured −52,978 B / −0.20%; the merge added +kernels to both builds and did not change the relationship.) **There is no size regression** — `static_plugin` is marginally smaller. Both configurations compiled cleanly with every reduced-size flag, which empirically confirms the analysis above. The plugin build plausibly comes out ahead @@ -631,35 +634,70 @@ places the plugin path could plausibly cost something: - **compute-bound** — 16 chained `[512, 512]` `MatMul`s. Host-side per-node cost is negligible against the GPU work. -Three runs per model per build; the table reports the median of the three per-run P50 values. +Three runs per model per build. **The two builds must be sampled interleaved** — one round is +`plugin/dispatch, plugin/compute, lib/dispatch, lib/compute`, repeated — rather than all samples of one build +followed by all samples of the other. This machine drifts: a build that has just finished, a warm browser, or a +warm GPU shifts timings by more than the effect being measured, and sequential sampling aliases that drift onto +the build under test. The sequential methodology used for the first pass understated a real regression by nearly +an order of magnitude (see below). The table reports the median of the three per-run P50 values. | Metric | `static_lib` | `static_plugin` | Delta | | --- | ---: | ---: | ---: | -| dispatch-bound, per-run P50 | 10.90 ms | 11.80 ms | +0.9 ms (+8%) | -| compute-bound, per-run P50 | 7.00 ms | 7.30 ms | +0.3 ms (+4%, within noise) | -| session init, dispatch model | 1136 ms | 1172 ms | +36 ms (+3%) | -| session init, compute model | 1250 ms | 1290 ms | +40 ms (+3%) | -| `suite0` net test time | 44.95 / 45.28 s | 45.71 / 44.93 s | none measurable | +| dispatch-bound, per-run P50 | 11.20 ms | 12.00 ms | +0.8 ms (+7%) | +| compute-bound, per-run P50 | 7.70 ms | 7.80 ms | +0.1 ms (within noise) | Reading of these numbers: -- **Execution is not measurably slower in any realistic case.** The compute-bound difference is inside run-to-run - variance, and `suite0`'s net test time — 2152 models, two samples per build — is indistinguishable between the - two builds. -- **A dispatch-bound graph pays roughly 8%.** That is the cost of routing per-node kernel work through the C API - adapters instead of a direct in-process virtual call. It only shows up when nodes do essentially no GPU work, - which is the worst case by construction rather than a representative one. Amortized over ~300 nodes it is on the - order of 3 µs per node. -- **Session creation costs about 3%.** This covers plugin EP registration, `GetCapability` and kernel creation - through the C API. Note the absolute figure is dominated by fetching and parsing the model, so the true relative - cost of the plugin machinery within session creation is higher than 3% — but it is tens of milliseconds once, - not per inference. - -A caution on methodology: the *total* wall time the test runner prints for `suite0` is not a usable metric. It -varied between 3 min 15 s and 4 min 04 s for the **same** `static_lib` binary across two runs, and `static_plugin` -produced both 4 min 13 s and 3 min 16 s. Only the second figure it prints — net test time, excluding the -per-model `before all` session-creation hooks — is stable enough to compare, and by that measure the two builds -are equal. An early single-sample comparison of the total figure suggested a 29% regression that does not exist. +- **Execution is not measurably slower once GPU work dominates.** The compute-bound samples overlap + (`static_plugin` 7.80 / 7.80 / 7.40 against `static_lib` 7.90 / 7.00 / 7.70 — the plugin build's best sample + beats two of the three control samples), so the difference is not resolvable at this sample count. +- **A dispatch-bound graph pays roughly 7%.** That is the cost of routing per-node kernel work through the C API + adapters instead of a direct in-process virtual call. Unlike the compute-bound case this one does separate + cleanly — the plugin build's slowest control sample is still faster than its own fastest sample. It only shows + up when nodes do essentially no GPU work, which is the worst case by construction rather than a representative + one. Amortized over ~300 nodes it is on the order of 3 µs per node. +- **Session creation costs about 3%** (measured pre-merge: 1136 → 1172 ms and 1250 → 1290 ms for the two models). + This covers plugin EP registration, `GetCapability` and kernel creation through the C API. The absolute figure + is dominated by fetching and parsing the model, so the true relative cost of the plugin machinery within session + creation is higher than 3% — but it is tens of milliseconds once, not per inference. + +Two cautions on methodology: + +- The *total* wall time the test runner prints for `suite0` is not a usable metric. It varied between 3 min 15 s + and 4 min 04 s for the **same** `static_lib` binary across two runs. Only the second figure it prints — net test + time, excluding the per-model `before all` session-creation hooks — is stable enough to compare, and by that + measure the two builds are equal. An early single-sample comparison of the total figure suggested a 29% + regression that does not exist. +- Timings taken with `--webgpu.profiling.mode=default` are inflated by the timestamp queries and must never be + compared against timings taken without it. + +#### The shared allocator must defer zero-initialize submission during a run + +Interleaved sampling exposed a genuine, code-level regression in the plugin path that sequential sampling had +hidden: the compute-bound model ran at 9.90 ms against the control's 7.30 ms, **+36%**, with no overlap between +the two sample sets. That the dispatch-bound model — 300 nodes against the compute model's 16 — did *not* show a +proportional cost ruled out a fixed per-node or per-run overhead and pointed at something scaling with bytes +allocated rather than with node count. + +The cause was buffer zero-initialization. `BufferManager::Create` clears a buffer recycled from the cache, and +`GpuBufferAllocator` takes a `should_submit_zero_initialize` predicate deciding whether that clear is submitted +before `Create` returns. During `Session::Run` it must **not** be: the clear is encoded into the run's own command +encoder and rides along with the run's submission, preserving dispatch batching. Outside a run there is no +upcoming submission to piggyback on, so the clear must be submitted immediately. `WebGpuExecutionProvider` and the +plugin factory's per-EP allocator both passed `!IsRunActive()`, but the factory's **shared** allocator — the one +backing intermediate tensors — hardcoded `true`, forcing a queue submit for every intermediate buffer mid-run. +The compute model allocates sixteen 1 MB intermediates, so it paid sixteen forced submits per inference. + +The shared allocator is created by `CreateAllocatorImpl` before any EP instance exists, so it had no EP to ask — +which is why the predicate was stubbed out rather than wired up. The fix moves the run-active flag onto +`WebGpuContext`, which is both reachable from the allocator and the more correct owner: it holds +`current_command_encoder_` / `current_compute_pass_encoder_`, the very state that batching depends on. +`WebGpuExecutionProvider` sets it in `OnRunStart` / `OnRunEnd` and its `IsRunActive()` now delegates to the +context, so all three allocator sites read one source of truth. With the fix the compute-bound gap disappears into +noise, as the table above shows. + +This bug predates the ORT Web migration and affects any consumer of the WebGPU plugin EP, not just the web build; +it is a candidate to land separately from the static-plugin work. From afd96be4d2d5bfffef34f40f4da87d0027609342 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Mon, 31 Aug 2026 18:46:13 -0700 Subject: [PATCH 032/107] Record why the shared allocator defect was missed in the pre-merge measurements Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../static_plugin_ep_registration_design.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md index 136b41eafc8b5..41e2f6ed0b062 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -699,6 +699,16 @@ noise, as the table above shows. This bug predates the ORT Web migration and affects any consumer of the WebGPU plugin EP, not just the web build; it is a candidate to land separately from the static-plugin work. +It was present in the pre-merge measurements too, which reported only +4% for this model. The reason is that the +cost is state-dependent rather than constant: `BufferManager::Create` only clears — and therefore only flushes — +a buffer it recycles from its cache, so the penalty depends on cache occupancy and appears as a wide, roughly +bimodal distribution rather than a fixed offset. Sequential sampling of the *same* defective binary produced +per-run P50s spanning 7.70 ms to 10.20 ms, a range that brackets the pre-merge figure. The merge itself did not +introduce or amplify the defect: of the three merged commits touching this path, #32272 only adds an early return +when an allocator is already registered (not reached with a single device), and #32269's subgroup-matrix MatMul is +gated on the F16 8x16x16 config, so the f32 compute-bound model never takes it. The lesson is about sample count +and interleaving on a high-variance metric, not about the merge. + > Should static factories be registered before environment creation or through environment construction options? From a75b4c129417d91f9d1e73a9d305236ef327a10f Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Tue, 1 Sep 2026 17:20:32 -0700 Subject: [PATCH 033/107] docs: add WebGPU static plugin EP design Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- ...boundary_and_web_integration_workstream.md | 5 +- .../static_plugin_ep_registration_design.md | 717 ++++++++++++++++++ .../static_plugin_ep_work_log.md | 521 +++++++++++++ .../webgpu_ep_extraction.md | 3 + 4 files changed, 1245 insertions(+), 1 deletion(-) create mode 100644 docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md create mode 100644 docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md index 2ff7f53894775..8dcbba70d3615 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md @@ -46,6 +46,8 @@ The design must address: The facility must be generic and validated with at least one non-WebGPU test plugin where practical. +The detailed design is in [Static Plugin EP Registration](static_plugin_ep_registration_design.md). + ## Process-global ownership and teardown Static registration removes the dynamic-library unload boundary. The current WebGPU plugin cleanup path cannot be @@ -176,5 +178,6 @@ The first five packages can proceed largely in parallel. Legacy-path removal wai ## Open questions -- Should static factories be registered before environment creation or through environment construction options? +- Resolved: neither. ORT core registers static factories during `OrtEnv` creation, after the environment is + constructed and published. See [Static Plugin EP Registration](static_plugin_ep_registration_design.md). - What is the stable representation of JavaScript-owned WebGPU objects at the C API boundary? diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md new file mode 100644 index 0000000000000..41e2f6ed0b062 --- /dev/null +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -0,0 +1,717 @@ +# Static Plugin EP Registration + +Status: Detailed design supporting +[Plugin Boundary and Web/Wasm Integration](plugin_boundary_and_web_integration_workstream.md). + +## Purpose + +Define how a plugin execution provider (EP) that is linked into the host binary is registered with ONNX Runtime, so +that statically linked and dynamically loaded plugin EPs share one provider implementation and one public API +boundary. + +WebGPU is the first consumer. The immediate deliverable is to move the statically linked WebGPU build off +`EpLibraryInternal` and onto the plugin EP path. + +## Background + +ORT has two `EpLibrary` implementations today: + +- `EpLibraryPlugin` (`onnxruntime/core/session/plugin_ep/ep_library_plugin.cc`) loads a shared library, resolves the + `CreateEpFactories` and `ReleaseEpFactory` symbols, and drives the factory lifecycle. +- `EpLibraryInternal` (`onnxruntime/core/session/plugin_ep/ep_library_internal.cc`) wraps an in-tree + `IExecutionProvider`, including `EpLibraryInternal::CreateWebGpuEp`. + +There is no `EpLibrary` that accepts factory entry points directly, so a provider compiled into the host cannot reach +the plugin path. + +This blocks ORT Web specifically. `cmake/onnxruntime_providers_webgpu.cmake` raises a `FATAL_ERROR` for the WebGPU +shared-module build under Emscripten, so static linking is the only way a Wasm build can ever reach the plugin +boundary. + +## Goals + +- Register a plugin EP that is linked into the host, reusing the existing plugin registration path after the point + where symbol lookup would occur. +- Present statically linked plugin code with the same runtime conditions as dynamically loaded plugin code. +- Require no change to provider sources between the two linkage modes. +- Avoid new host call sites for in-tree providers. +- Keep the shared-library path byte-for-byte unchanged at runtime. + +## Non-goals + +- A public C API for registering static plugin EPs. Deferred until a provider is genuinely out-of-tree. +- Removal of the EP API adapters. Tracked separately. +- Removal of the direct `IExecutionProvider` WebGPU path. That is the final step of the workstream, not this change. +- Support for static plugin EPs in minimal builds. `cmake/onnxruntime_session.cmake` excludes all of + `core/session/plugin_ep/` when `onnxruntime_MINIMAL_BUILD` is set. + +## Decisions + +### D1: Registration mechanism is a link-time registry in ORT core + +ORT core holds a hand-written, `#if`-guarded list of statically linked plugin EP entry points, mirroring the shape of +`EpLibraryInternal::CreateInternalEps`. A new `EpLibraryStaticPlugin` accepts the entry points directly and reuses +`EpLibraryPlugin`'s factory lifecycle logic without the dynamic-library load and unload steps. + +Rationale: every host — `onnxruntime_test_all`, the Python bindings, `onnxruntime.dll`, and Wasm — gets the provider +with no per-host call site. + +Alternatives considered: + +- CMake-generated registry using an X-macro. Rejected: added build-system machinery for generality that may never be + needed while providers remain in-tree. +- A public `RegisterStaticExecutionProviderLibrary` C API called by each host. Rejected for now: it requires a call + site in every statically linked host and introduces a window in which `GetEpDevices` reports no devices. Revisit + when an out-of-tree provider needs static linking. + +### D2: Entry point names are macro-parameterized + +`onnxruntime/core/providers/webgpu/ep/api.cc` is parameterized by `ORT_PLUGIN_EP_ENTRY_POINT_PREFIX`. + +The shared build must continue to export the unprefixed `CreateEpFactories` and `ReleaseEpFactory`, because +`EpLibraryPlugin::Load` resolves those exact names. The static build emits prefixed variants, for example +`WebGpu_CreateEpFactories`, so that multiple static plugins can coexist in one binary. + +### D3: No new EP library ABI hook for teardown + +All teardown remains in `ReleaseEpFactory`. Process-global shutdown that belongs to the host — currently +`google::protobuf::ShutdownProtobufLibrary()` — is compiled out under `ORT_PLUGIN_EP_OWNS_PROCESS_GLOBALS`. + +Rationale: a new optional entry point cannot be relied upon, because an older ORT would load a newer plugin and +silently ignore it. Behavior that varies with the host version is worse than a compile-time decision. + +The legacy static WebGPU cleanup block in `onnxruntime/core/session/ort_env.cc` is already guarded by +`defined(USE_WEBGPU) && !defined(ORT_USE_EP_API_ADAPTERS)`. The static plugin configuration defines +`ORT_USE_EP_API_ADAPTERS`, so the block compiles out on its own and needs no edit. It must be kept for the default +internal-EP WebGPU build. + +Consequence to test for: static linking has no library unload, so provider process-global state survives +unregistration. Registering, running, unregistering, re-registering, and running again is the primary regression case. + +### D4: One new durable build option + +- `onnxruntime_USE_EP_API_ADAPTERS` is unchanged. It is transitional and is retired when WebGPU moves fully onto the + public plugin API. +- `onnxruntime_WEBGPU_STATIC_PLUGIN` is new and durable. It selects linkage into the host rather than a loadable + module. +- `onnxruntime_WEBGPU_LINKED_INTO_HOST` is derived and used at the linkage sites. + +`onnxruntime_USE_EP_API_ADAPTERS` currently conflates three meanings: compiling against the adapters, building a +separate module, and registering by path in tests. The static plugin configuration answers the first yes and the +other two no, so only the linkage and test sites move to the derived option. The global `add_compile_definitions` in +`cmake/CMakeLists.txt` stays as-is, and no provider sources change. + +`cmake/onnxruntime_providers_webgpu.cmake` gains a third arm. The Emscripten and `onnxruntime_BUILD_CACHE` +`FATAL_ERROR`s narrow to the shared-module case, which is what unblocks ORT Web. + +`build.py` gains `--use_webgpu static_plugin`. + +`onnxruntime_WEBGPU_STATIC_PLUGIN` validates its prerequisites at configure time and fails with `FATAL_ERROR` if +they are not met: it requires `onnxruntime_USE_WEBGPU` and `onnxruntime_USE_EP_API_ADAPTERS`, and it rejects a +minimal build. The last one is a temporary guard rather than a permanent restriction. A minimal build excludes +`core/session/plugin_ep` from `onnxruntime_session_srcs`, and `Environment::CreateAndRegisterStaticPluginEps` is +compiled out with it, but `cmake/onnxruntime_providers_webgpu.cmake` still builds and links the provider. The +combination would therefore produce a binary containing the WebGPU EP that never registers it, with nothing +reporting the problem at configure, build or run time. Allowing minimal builds to use a static plugin EP is +worthwhile and needs the registration path to be available there first; the exclusion is currently justified by +provider-bridge dependencies, and only two of the files under `plugin_ep` are provider-bridge, so the subset may be +separable. Until then a loud configure error is preferable to a silently EP-less binary. + +The CUDA plugin is unaffected: it is gated by `onnxruntime_BUILD_CUDA_EP_AS_PLUGIN` and sets +`ORT_USE_EP_API_ADAPTERS` as a private target compile definition, never referencing the CMake option. CUDA's split +between role (`BUILD_CUDA_EP_AS_PLUGIN`) and mechanism (`ORT_USE_EP_API_ADAPTERS`) is the in-tree template for +WebGPU's eventual cleanup. + +### D5: Test infrastructure treats the library path as optional + +In `onnxruntime/test/unittest_util/test_dynamic_plugin_ep.cc`, `InitializationConfig::ep_library_path` becomes +optional. When absent, library registration is skipped because ORT core has already performed it, and the RAII +registration handle is left empty. The existing handle deleter already tolerates an empty handle. Device selection, +de-duplication, and factory creation are unchanged. + +Virtual devices are enabled through the `allow_virtual_devices` environment configuration entry +(`kOrtEnvAllowVirtualDevices`) supplied at environment creation, rather than the `.virtual` registration-name suffix. +The suffix is unavailable because ORT core chooses the registration name for statically linked providers. + +The `dynamic_plugin_ep_infra` naming becomes inaccurate. Renaming is deferred to a separate mechanical change. + +### D6: Static plugin EPs are registered after the environment is published + +This is the subtlest decision and the one that constrains the others. + +**Problem.** The natural insertion point is next to `CreateAndRegisterInternalEps` inside `Environment::Initialize`. +Registration enumerates devices, which calls provider code: + +``` +OrtEnv::GetOrCreateInstance() + lock(m_) non-recursive + Environment::Create() -> Initialize() + Environment::CreateAndRegisterStaticPluginEps() + Environment::RegisterExecutionProviderLibrary() + EpInfo::Create -> factory GetSupportedDevices() + webgpu::ep::Factory::GetSupportedDevices() provider code + Api().ep.GetEnvConfigEntries() + OrtEnv::TryGetInstance() + lock(m_) self-deadlock +``` + +`Environment::CreateAndRegisterInternalEps` already documents this hazard, and the internal WebGPU factory avoids it +by capturing `allow_virtual_devices` at construction. A plugin factory cannot use that workaround, because the public +API is all it has. + +The problem is not confined to one function. Plugin EPs receive the full `OrtApi`, so any API that resolves the +environment is affected, including `CreateEnv`, which re-enters `GetOrCreateInstance` and deadlocks on the same mutex. + +**Decision.** `OrtEnv::GetOrCreateInstance` publishes `p_instance_` and takes its own reference *before* invoking +`Environment::CreateAndRegisterStaticPluginEps`, and `OrtEnv::m_` becomes a `std::recursive_mutex`. Registration +therefore runs against a fully constructed and published environment. + +Properties: + +- The entire `OrtApi` behaves normally for provider code. `TryGetInstance` finds a published instance, and a + re-entrant `CreateEnv` acquires the recursive lock and increments the reference count. +- No new contract for provider authors, and nothing for a future API author to remember. +- No race. Other threads block on `m_` for the duration, which is already true across `Environment::Create`. +- The reference count cannot reach zero mid-construction, because the creating thread's reference is taken first. +- Provider sources need no conditional compilation for the two linkage modes. + +Costs and limits: + +- Failure during registration requires explicit teardown of the just-published instance. +- A recursive mutex is normally undesirable. Here it states the actual invariant: the environment creation path can + legitimately re-enter the environment accessor on the same thread. +- Callers of `onnxruntime::Environment::Create` do not get static plugin EPs. In the tree this is one production call + site plus tests and orttraining sample binaries. +- Provider code that starts a thread which calls `CreateEnv` and then joins it will deadlock. This is already true of + any code running inside `Environment::Create`. + +The call to `Environment::CreateAndRegisterStaticPluginEps` therefore lives in `ort_env.cc`. Static plugin +registration is a process-singleton concern tied to the `OrtEnv` lifetime, so the singleton wrapper is its correct +home. + +Alternatives considered: + +- Thread-local pointer to the environment under construction, consulted by `GetEnvConfigEntries`. Rejected: it + addresses one function, while the reachable surface is the whole `OrtApi`. +- Defer device enumeration until after environment creation. Rejected: lazy initialization moves registration + failures from environment creation to the first `GetEpDevices` call, which is a worse place to report them. +- Register outside the lock after publishing. Rejected: another thread can observe a published environment whose + static EPs are not yet registered, and avoiding that requires a second lock and a completion barrier. + +`onnxruntime::Environment` remains directly constructible, and tests rely on that. `OrtEnv::GetOrCreateInstance` +returns any existing instance and ignores the logging manager, threading options, and configuration entries passed by +later callers, so a test needing specific threading or logging configuration must bypass it. Those tests are +deliberately opting out of process-singleton semantics and should not receive static plugin EPs. + +### D7: The statically linked build does not use manual C++ API initialization + +`include/onnxruntime/ep/api.h` force-enables `ORT_API_MANUAL_INIT` around its include of `onnxruntime_cxx_api.h`, and +`onnxruntime::ep::ApiInit` calls `Ort::InitApi(ort_api)`. That is required for a plugin EP shared library, which must +not call `OrtGetApiBase()` itself. + +`onnxruntime_cxx_api.h` emits `#pragma detect_mismatch("ORT_API_MANUAL_INIT", ...)` on MSVC, so every translation +unit linked into one binary must agree. The statically linked plugin EP is linked with ORT core and with test code, +neither of which uses manual initialization, so forcing it on produces `LNK2038` for every EP object file. + +The build therefore defines `ORT_PLUGIN_EP_STATICALLY_LINKED` on the statically linked plugin EP target. Under that +macro `ep/api.h` includes `onnxruntime_cxx_api.h` unmodified and `ApiInit` skips `Ort::InitApi`. The C++ API then +default-initializes from `OrtGetApiBase()->GetApi(ORT_API_VERSION)`, which resolves in-process and yields the same +`OrtApi` that `ApiInit` would have installed, because the EP and ORT are the same binary and therefore the same +version. `onnxruntime::ep::ApiPtrs` is still populated from the `OrtApiBase*` that ORT passes to `CreateEpFactories`, +so the EP's own API access is unchanged. + +The macro is generic rather than WebGPU-specific because `ep/api.h` is shared with the CUDA plugin EP. + +## Implementation order + +1. `EpLibraryStaticPlugin`, factoring the shared factory lifecycle out of `EpLibraryPlugin`. +2. The static registry and `Environment::CreateAndRegisterStaticPluginEps`, with the `ort_env.cc` + publish-then-register change. +3. `ep/api.cc` entry point prefixing and host-owned process-global guards. +4. Build option and `build.py` changes. +5. Test infrastructure changes. +6. `ORT_PLUGIN_EP_STATICALLY_LINKED` in `include/onnxruntime/ep/api.h`, per D7. +7. Applying the WebGPU plugin EP test definitions to `onnxruntime_provider_test` as well as `onnxruntime_test_all`, + via a shared `onnxruntime_set_webgpu_plugin_ep_test_definitions` cmake function. `onnxruntime_provider_test` holds + the operator tests, so without this the plugin path has no operator test coverage. It also needs + `ORT_UNIT_TEST_HAS_WEBGPU_STATIC_PLUGIN_EP` in its own right, because that macro is what enables the + `allow_virtual_devices` environment configuration entry at environment creation in `test_main.cc`. + +## Follow-up cleanup + +`Environment::CreateAndRegisterInternalEps` stays where it is, at the end of `Environment::Initialize`. Internal EP +factories are ORT-core code that does not re-enter the `OrtEnv` singleton, so they are not subject to the constraint +in D6. Moving them would also remove EP devices from every directly constructed `Environment`, and `InferenceSession` +reads `Environment::GetOrtEpDevices` directly. For static plugin EPs the resulting gap is unimplemented new +functionality; for internal EPs it would be a silent regression. + +The `allow_virtual_devices` parameter threaded through `EpLibraryInternal::CreateInternalEps` exists only for the +internal WebGPU EP factory, which cannot query the environment at `GetSupportedDevices` time. The comment in +`CreateAndRegisterInternalEps` documents that constraint. Both can be deleted once WebGPU is no longer an internal EP: +the remaining internal EPs are CPU, kept deliberately as a special case, and DML, which is no longer maintained. +Neither takes the parameter. In a build that defines `ORT_USE_EP_API_ADAPTERS` this is already true, since the +internal WebGPU EP is compiled out and the parameter reaches `ORT_UNUSED_PARAMETER`. + +This does not affect the `.virtual` registration-name suffix or the `allow_virtual_devices` environment +configuration entry, which remain in use for dynamically registered plugin libraries. + +## Risks and open questions + +- Whether adapter-compiled WebGPU objects link into ORT core without duplicate-symbol or ODR problems. One real + instance was found and resolved: the `ORT_API_MANUAL_INIT` `detect_mismatch` guard, see D7. Otherwise resolved: + `onnxruntime_test_all` and `onnxruntime_provider_test` both link cleanly against the statically linked EP. +- Provider process-global state surviving unregistration, since static linking has no unload event. Covered by the + re-registration regression case in D3. Not yet exercised: `OrtEnv` is a reference-counted process singleton and the + unit test main holds a reference for the lifetime of the process, so a second `Environment` construction (and + therefore a second `CreateEpFactories` call) is not reachable from the existing test binaries. This needs either a + dedicated single-test process or a test that fully releases and recreates the environment. +- Whether any shipped ORT Web configuration uses an extended-minimal build, which excludes plugin EP infrastructure + entirely. +- Whether any `std::condition_variable` is paired with `OrtEnv::m_`, which would require `condition_variable_any` + under a recursive mutex. +- The existing rejection of `onnxruntime_BUILD_DAWN_SHARED_LIBRARY` together with the adapters. Initially left + rejecting both plugin kinds. +- Binary size and dead-code elimination for statically linked providers, per the workstream completion criteria. + +## Validation status + +Verified on Windows with `--use_webgpu static_plugin`, Debug: + +- `onnxruntime_test_all` and `onnxruntime_provider_test` build and link. +- The statically linked WebGPU plugin EP is registered by ORT core during `Ort::Env` creation, and its `OrtEpDevice` + is enumerated and selectable, confirming the D6 publish-then-register ordering resolves the self-deadlock. +- `InferenceSessionTests.WebGpuVirtualDeviceCompileOnlyEndToEnd` and + `InferenceSessionTests.WebGpuVirtualDeviceRejectedWithoutCompileOnly` pass, which covers session creation and EP + instantiation end to end through the public V2 API on the static plugin path. + +Operator execution was initially not verifiable locally: the original development machine was a virtual machine with +no real GPU, so only the virtual WebGPU device was surfaced and non-compile-only sessions were rejected up front. This +was confirmed to be a property of the host rather than of static linking by building the shared library plugin +configuration (`--use_webgpu shared_lib`) on the same host and running the same tests: it passed and failed exactly +the same set (`WebGpuVirtualDeviceCompileOnlyEndToEnd` and `WebGpuVirtualDeviceRejectedWithoutCompileOnly` pass; +`WebGpuCompileOnlySkipsFinalization` and `TestStrictShapeInference` fail with the identical virtual-device rejection +message). + +This gap has since been closed on a machine with a real GPU (NVIDIA RTX 5060 Ti). The `static_plugin` and internal-EP +configurations were built from the same tree, RelWithDebInfo, differing only in the EP path, and their test results +compared: + +- `onnxruntime_provider_test`: 0 failures in both. **Across the 5894 tests common to both builds there were zero + status differences.** +- `onnxruntime_test_all`: 1965 passed in both, no differences. All three `InferenceSessionTests.WebGpu*` pass on the + static plugin build, including `WebGpuCompileOnlySkipsFinalization`, confirming its earlier failure was indeed a + property of the GPU-less host. + +### Known test coverage difference on the plugin path + +`cmake/onnxruntime_unittests.cmake` gates the `test/providers/webgpu/*` sources on +`onnxruntime_USE_WEBGPU AND NOT onnxruntime_USE_EP_API_ADAPTERS`, so those tests are **absent** from any adapter +build, including `static_plugin`. They are white-box tests that include internal WebGPU EP headers and therefore +cannot compile against the adapter boundary by construction. + +Of the 44 tests this excludes, 16 are `DISABLED_` and never run, and most of the rest assert on EP internals +(`WebGpuContextTest`, `ActivationCacheKeyTest`). However, 12 of them — `HardSwish_WebGPU` and `MatMul2BitsWebGpu` — +are genuine operator tests, and they currently run **only** on the internal EP path. This is a known coverage gap of +the plugin path. Closing it would mean rewriting them as EP-agnostic operator tests so both paths execute them; that +is deliberately left out of this change to keep it focused. + +Separately, the `InferenceSessionTests.WebGpu*` virtual-device tests are deliberately *paired* rather than shared +between the two builds, since each drives a different factory. `WebGpuVirtualDeviceCompileOnlyEndToEnd` and +`WebGpuVirtualDeviceRejectedWithoutCompileOnly` are the plugin-build counterparts of `WebGpuEpFactoryVirtualDevice` +and `WebGpuEpFactoryRejectsVirtualDeviceWithoutCompileOnly`. Only `WebGpuCompileOnlyUsesNoOpAllocator` has no +counterpart, because it asserts an internal allocator *type* through `dynamic_cast`, which is inherently +unobservable across the ABI boundary. + +The shared library plugin configuration was also re-verified after the D7 change, since `include/onnxruntime/ep/api.h` +is shared with the CUDA plugin EP. It builds, and `onnxruntime_providers_webgpu.dll` still exports exactly the +unprefixed `CreateEpFactories` and `ReleaseEpFactory` required by `EpLibraryPlugin::Load`, per D2. + +### CI coverage + +No pre-existing CI leg built this configuration, since both new CMake options default to `OFF`. Two jobs were added: + +- `webgpu_static_plugin_build_x64_RelWithDebInfo` in `.github/workflows/windows_webgpu.yml`, which builds and runs the + tests on a GPU-equipped pool. This closes the operator execution gap above. Note that it must not set + `onnxruntime_BUILD_DAWN_SHARED_LIBRARY=ON`, which is incompatible with `onnxruntime_USE_EP_API_ADAPTERS`. +- `build-linux-webgpu-static-plugin-x64-release` in `.github/workflows/linux_webgpu.yml`, build only, mirroring its + sibling job. This is the only GCC coverage of the EP adapters, and the only coverage of the `onnxruntime_python.cmake` + change, hence `--build_wheel` is kept. + +The first run of the Linux job surfaced `-Werror=maybe-uninitialized` errors. The EP adapter `OpKernelInfo` defines its +attribute accessors inline in the header, so GCC can see their failure and exception paths, whereas the in-tree +`OpKernelInfo` hides them behind an out-of-line definition. Any value that is only read when the attribute lookup +succeeded must be value-initialized. + +More generally, this job is the first time the EP adapters have ever been compiled with GCC, and it has surfaced +several successive batches of `-Werror` diagnostics — including false positives inside `absl::InlinedVector` and, in +optimized builds, `-Werror=array-bounds` on `std::string` concatenation chains that the adapter headers cause to be +inlined more aggressively. Because ninja stops at the first failure and reports only the targets already in flight, +**each CI run reveals just one batch**. When touching this configuration, enumerate the full set locally in a single +pass — an AlmaLinux 8 container with `gcc-toolset-14` reproduces the CI compiler exactly — rather than discovering +them one CI round trip at a time. Fixes that are correct on their own merits are applied at the source; irreducible +false positives use a suppression scoped to the `onnxruntime_providers_webgpu` target, GCC only, and conditional on +`onnxruntime_USE_EP_API_ADAPTERS`, so non-adapter builds stay strict. This mirrors what the CUDA plugin EP does in +`cmake/onnxruntime_providers_cuda_plugin.cmake`. + +That full local enumeration has since been done, and it is worth recording what it found: of the ~340 warnings GCC 14 +emits across the whole `static_plugin` build, **exactly one is attributable to ORT's own source** — the +`-Warray-bounds` false positive in `contrib_ops/webgpu/quantization/matmul_nbits_mlp.cc`, fixed by building the +expression with `MakeString` instead of a `std::string` concat chain. Everything else lives in Dawn, ONNX, +flatbuffers, or the telemetry dependency, in targets that do not use `-Werror`. The practical lesson is that the +adapters do not make ORT's code broadly warning-prone; they perturb inlining at a small number of specific sites, +so a targeted source fix is almost always available and preferable to a blanket target-wide suppression. + +When validating such a fix locally, always rebuild the *unfixed* source through the same harness first and confirm it +still errors. A harness that silently fails to exercise the warning is indistinguishable from a successful fix. + +### Emscripten and ORT Web + +Making the statically linked WebGPU EP reachable through the plugin path is what motivates this design, so the +Emscripten configuration was built before landing the change: `--build_wasm --enable_wasm_simd +--enable_wasm_threads --use_webgpu static_plugin --use_webnn --target onnxruntime_webassembly`. It configures, +compiles and links cleanly, producing `ort-wasm-simd-threaded.asyncify.{mjs,wasm}`. + +`static_plugin` is in fact the only plugin kind Emscripten supports — `cmake/onnxruntime_providers_webgpu.cmake` +raises a `FATAL_ERROR` for `shared_lib` there, since there is no runtime library to load. + +No CI leg covers this yet. The WASM WebGPU jobs pass a bare `--use_webgpu`, which defaults to `static_lib` +(`tools/ci_build/build_args.py` uses `const="static_lib"`), so they still build the internal EP. This change makes +the configuration available to ORT Web; switching ORT Web over to it is separate work, and has prerequisites that +this change does not address. See [ORT Web migration](#ort-web-migration). + +A green build is *not* sufficient evidence that the registration is live. `CreateStaticPluginEpLibraries()` in +`onnxruntime/core/session/plugin_ep/ep_static_plugins.cc` is guarded by +`#if defined(USE_WEBGPU) && defined(ORT_WEBGPU_STATIC_PLUGIN)`; if that define does not reach ORT core the function +still compiles, simply returning an empty vector, and the build is just as green while registering nothing. The +check that actually discriminates is at the symbol level — ORT core must hold an *undefined* reference to the +prefixed entry points, resolved by the provider library: + +``` +llvm-nm libonnxruntime_session.a | grep WebGpu_ # U WebGpu_CreateEpFactories / U WebGpu_ReleaseEpFactory +llvm-nm libonnxruntime_providers_webgpu.a | grep WebGpu_ # T WebGpu_CreateEpFactories / T WebGpu_ReleaseEpFactory +``` + +Both were observed on the Emscripten build, which confirms the guard was live, the D3 entry-point prefixing worked, +and the link resolved core's reference against the provider. That check validates build, link and registration +wiring; runtime behaviour was established separately by the ORT Web prototype described under +[ORT Web migration](#ort-web-migration). + +A practical build note: the WASM build needs `node` on `PATH` — `node_helper.cmake` does a hard `find_program` and +fails configure without it. emsdk bundles a suitable one under `cmake/external/emsdk/node//bin`. + +## ORT Web migration + +The intended end state is that the WebGPU EP is always a plugin EP and `--use_webgpu static_lib` goes away +entirely, ORT Web included. Building the EP into the WASM binary as a static plugin, which this change enables, is +only the first half of that. The second half is *selection*, and two gaps stand between the current state and a +working ORT Web on the plugin path. Neither is visible to the linker, so the symbol-level evidence above does not +speak to either. + +Both gaps have since been closed by a throwaway prototype, which ran the full `js/web` WebGPU operator suite in +Edge against a WASM `static_plugin` binary: 2152 tests, all passing. The design below is therefore validated end +to end rather than merely plausible; what remains is productionization, not discovery. The prototype's findings are +recorded inline in the steps that follow. + +**ORT Web selects the EP by name, and that name resolves only to the internal EP.** +`js/web/lib/wasm/session-options.ts` maps the `'webgpu'` backend to `epName = 'WebGPU'`, which reaches +`OrtAppendExecutionProvider` in `onnxruntime/wasm/api.cc` and from there +`OrtApis::SessionOptionsAppendExecutionProvider`. That API is a closed table of hardcoded EP names in +`onnxruntime/core/session/provider_registration.cc`, and its `EpID::WebGPU` case is compiled out under +`ORT_USE_EP_API_ADAPTERS`. On a `static_plugin` build the EP therefore registers at `OrtEnv` creation and is then +unselectable from JS: the call returns "WebGPU execution provider is not supported in this build". This is by +design rather than an oversight — the name-based API is the internal-EP mechanism, and plugin EPs are selected +through `RegisterExecutionProviderLibrary` plus `OrtEpDevice`-based selection — but it does mean ORT Web needs new +plumbing rather than a recompile. + +**No WebGPU `OrtEpDevice` exists under Emscripten.** Emscripten was neither `WIN32`, `LINUX` nor `APPLE`, so +`cmake/onnxruntime_common.cmake` fell through to `core/platform/device_discovery_default.cc`, which discovers only +a CPU device. `Factory::GetSupportedDevicesImpl` in `onnxruntime/core/providers/webgpu/ep/factory.cc` creates an +`OrtEpDevice` only for devices of type `OrtHardwareDeviceType_GPU`. The one fallback, the virtual GPU device in the +same function, is opt-in through the `allow_virtual_devices` environment configuration entry and is deliberately +registered without allocator info, because it exists to back a device-free compile-only session. So even after +selection plumbing is added, `GetEpDevices` would offer nothing for JS to select. + +A workable order: + +1. Make WebGPU visible as a device under Emscripten by adding an Emscripten device discovery implementation. + `cmake/onnxruntime_common.cmake` gains an `Emscripten` arm selecting a new + `core/platform/emscripten/device_discovery.cc`, which reports the CPU device via `GetCpuDeviceFromCPUIDInfo()` + plus a GPU `OrtHardwareDevice` when a synchronous `EM_ASM_INT` check finds `navigator.gpu`. No change to + `Factory::GetSupportedDevicesImpl` is needed: its existing loop filters on `OrtHardwareDeviceType_GPU`, so a + discovered GPU flows through and picks up both allocator infos unmodified. + + The prototype first tried the other option — having the factory synthesize the device itself through + `CreateHardwareDevice` — and it worked, but it is the wrong layer. `GetSupportedDevices` documents its `devices` + argument as "the `OrtHardwareDevice` instances that are available", so that array is the source of truth, and + `CreateHardwareDevice` is documented for devices with no hardware behind them ("e.g., virtual"). A browser GPU + is real hardware that ORT's discovery layer merely failed to enumerate, so the producer is the place to fix it. + Leaving it in the factory would also oblige every future Emscripten plugin EP to repeat the same workaround. + + A capability check is a legitimate form of discovery here. `core/platform/apple/device_discovery.cc` sets the + precedent: it enumerates nothing, it asserts that one GPU exists on Apple Silicon and hardcodes the vendor id. + `navigator.gpu` is a stronger signal than that — an actual runtime capability probe, and the same one ORT Web's + `initEp` uses to decide whether WebGPU is usable. `EM_ASM` from ORT core is likewise established practice + (`core/graph/model.cc`, `core/framework/external_data_loader.cc`), not just an EP-layer habit. + + The honest limitation is that no adapter information is available. `_OrtInit` creates the `OrtEnv` and therefore + triggers discovery, and it runs before `navigator.gpu.requestAdapter()` in ORT Web's `initEp`, so `vendor_id` + and `device_id` are left at 0. That is acceptable: Apple reports no device id either, and browsers mask + `GPUAdapterInfo.vendor` for fingerprinting resistance regardless. The device is deliberately *not* tagged with + `kOrtHardwareDevice_MetadataKey_IsVirtual`, which is what keeps it distinct from the virtual device — that path + is allocator-less by design and cannot back a real session. + + A later change on `main` added a second fallback to `GetSupportedDevicesImpl` alongside the virtual device: when + no GPU device is discovered, `allow_software_adapter` (opt-in via the `kAllowSoftwareAdapterEnvironmentVariable` + environment variable) advertises WebGPU against the *CPU* `OrtHardwareDevice`. It does not change the analysis + above. Both upstream fallbacks work around a device the discovery layer failed to enumerate; they are opt-in + precisely because misreporting a device is not safe to do by default. Under Emscripten the device is genuinely + there and cheaply detectable, so fixing discovery keeps the default path correct and needs no opt-in. The + primary matching loop that our discovered GPU device flows through is unchanged by that work. + + One accepted side effect: `ep_library_internal.cc` creates an internal `WebGpuEpFactory` in a + `USE_WEBGPU && !ORT_USE_EP_API_ADAPTERS` build, so today's `static_lib` ORT Web build starts producing an + `OrtEpDevice` where it previously produced none. That build selects by name and is slated for removal, so the + change is benign. + +2. Expose plugin EP selection in the WASM C API. `g_env` is already available in `api.cc`, so `GetEpDevices` and + `SessionOptionsAppendExecutionProvider_V2` are both reachable. A narrow helper that appends by EP name and + resolves the device internally avoids marshalling arrays of device pointers across the JS boundary. The new + entry point needs to be exported from `cmake/onnxruntime_webassembly.cmake`. + + *Prototype result:* the name-resolving helper is the right shape. Note that exporting it takes three edits, not + one: `EMSCRIPTEN_KEEPALIVE` in `api.h`, the `JSPI_EXPORTS` list in `cmake/onnxruntime_webassembly.cmake` for + JSPI builds, and the `wrapAsyncAPIs` list in `onnxruntime/wasm/pre-async.js` for Asyncify builds. Omitting the + last one is the easy mistake, and Asyncify is the default variant. Reporting matters too: on no match the helper + should raise a status naming the EPs that *are* registered, which is what makes a misconfiguration diagnosable + from JS through `checkLastError`. + +3. Switch `js/web` to the new call. The `BUILD_DEFS.DISABLE_WEBGPU` branch that falls back to the JS EP is + unaffected. Confirm that the EP options ORT Web passes today — `deviceId`, `webgpuInstance`, `webgpuDevice`, + `preferredLayout` and the buffer cache modes — still reach the provider, since `Factory::CreateEpImpl` obtains + them from the session config options. + + *Prototype result:* the options do reach the provider unchanged, and the reason is worth recording. Both paths + derive their config prefix from `OrtSessionOptions::GetProviderOptionPrefix` keyed on the EP name, so V2 lands + the options under exactly the same `ep.webgpuexecutionprovider.*` keys the name-based path used. One JS change + is required though: `session-options.ts` passes the short name `'WebGPU'`, whereas `OrtEpDevice` selection + matches the canonical `'WebGpuExecutionProvider'` returned by `EpDevice_EpName`. + + The productionization question this raises is *conditionality*: is it safe to switch to V2 unconditionally + whenever WebGPU is enabled? It is, provided the WASM build flips at the same time. `BUILD_DEFS.DISABLE_WEBGPU` + is defined in `js/web/script/build.ts` as exactly `!USE_WEBGPU_EP`, so the bundle is already specialized per + WASM flavour by `--webgpu-ep` — the JSEP bundle takes the other branch entirely and is unaffected. Flipping the + WebGPU WASM builds to `static_plugin` (step 4) therefore makes `!DISABLE_WEBGPU` imply `static_plugin` by + construction, and no additional build define or runtime capability probe is needed. What is *not* safe is + landing the JS change without the build change, since the two must move together. + +4. Flip the ORT Web WASM builds to `static_plugin` and add a CI leg. The two builds are + `Build (simd + threads + WebGPU experimental)` and its JSPI sibling in + `.github/workflows/linux-wasm-ci-build-and-test-workflow.yml`, both of which pass a bare `--use_webgpu` and so + get `static_lib` from `build_args.py`'s `const="static_lib"`. Neither uses `--minimal_build`, so the + minimal-build `FATAL_ERROR` guard does not stand in the way of this migration. No pipeline builds WASM with + `static_plugin` today, so without a leg this path stays validated at the symbol level only. + + *Prototype note:* the `js/web` test runner cannot currently exercise this configuration end to end, because + `script/test-runner-cli.ts` spawns `script/build` with only `--bundle-mode` and does not forward `--webgpu-ep`. + Setting `npm_config_webgpu_ep` in the environment works as a stopgap, since `script/build.ts` reads it, but a CI + leg should forward the flag properly. Note also that without `--webgpu-ep` the `webgpu` backend silently routes + through JSEP instead, so a leg that omits it would pass while testing nothing relevant. + +### Interaction with the ORT Web reduced-size build options + +The ORT Web WASM builds are size-sensitive and pass a set of size-reduction flags that the local prototype builds +did not. `linux-wasm-ci-build-and-test-workflow.yml` applies `--disable_ml_ops --disable_generation_ops +--disable_types string float4 float8 optional sparsetensor --include_ops_by_config +onnxruntime/wasm/reduced_types.config --enable_reduced_operator_type_support`, and `web.yml` adds `--disable_rtti` +for the release job. Since the migration changes how the WebGPU EP is compiled and linked, each of these was +checked against the plugin path. + +**None of them need adapting.** The findings: + +- *Compile definitions reach the WebGPU target in every mode.* `onnxruntime_providers_webgpu` is created through + `onnxruntime_add_static_library` / `onnxruntime_add_shared_library_module`, both of which call + `onnxruntime_configure_target` → `onnxruntime_set_compile_flags`. That function is where every `DISABLE_*` and + `REDUCED_OPS_BUILD` definition is applied, so the `static_lib`, `static_plugin` and shared-library builds all + receive an identical set. There is no propagation gap introduced by the plugin boundary. + +- *Operator and type reduction never applied to WebGPU, before or after.* + `op_registration_utils.get_kernel_registration_files()` hardcodes the CPU registration files (plus CUDA when + requested); WebGPU is not in the list, so `--include_ops_by_config` does not rewrite WebGPU kernel + registrations. `onnxruntime/wasm/reduced_types.config` is `!no_ops_specified_means_all_ops_are_required` — it + performs global *type* reduction only, with no op exclusion — and that mechanism works through the + `op_kernel_type_control` macros, which neither `core/providers/webgpu` nor `contrib_ops/webgpu` uses. Its saving + is entirely CPU-EP-side and is unchanged by the plugin switch. This is a pre-existing gap, not a regression. + +- *The type and ML/generation op flags are inert for WebGPU.* No source under `core/providers/webgpu` or + `contrib_ops/webgpu` references `DISABLE_ML_OPS`, `DISABLE_GENERATION_OPS`, `DISABLE_SPARSE_TENSORS`, + `DISABLE_OPTIONAL_TYPE`, `DISABLE_FLOAT8_TYPES`, `DISABLE_FLOAT4_TYPES` or `DISABLE_STRING_TYPE`. + +- *`--disable_rtti` is safe on the plugin path.* Neither `include/onnxruntime/ep` (the EP API adapters) nor + `onnxruntime/core/session/plugin_ep` uses `dynamic_cast` or `typeid`. + +One consequence is worth recording for the *shared library* plugin EP, which is out of scope here but shares this +code. `onnxruntime_c_api.h` contains no `DISABLE_*` guards, so the `OrtApi` struct layout is invariant under these +flags and the ABI is stable. `onnxruntime_cxx_api.h` does guard some `Ort::Value` members (the sparse tensor +methods) — an EP DLL built with a different `--disable_types` than its host would therefore see a different C++ +header surface, but since those wrappers are inline over stable function pointers this is a source-compatibility +concern rather than an ABI one. For `static_plugin` everything is compiled in one tree with one set of defines, so +the question does not arise. + +**What is *not* covered by any existing flag** is the plugin machinery itself. Relative to `static_lib`, the +`static_plugin` build adds `core/providers/webgpu/ep/*` (excluded from the source list in the `static_lib` branch +of `onnxruntime_providers_webgpu.cmake`), inlines the header-only EP API adapters into every WebGPU translation +unit, and reaches the `OrtGraph` / `OrtNode` C API graph views through `OrtEp::GetCapability`. Note that +`core/session/plugin_ep` is *not* part of that delta: `onnxruntime_session.cmake` only excludes it for +`onnxruntime_MINIMAL_BUILD`, so today's WASM builds already compile it in. + +That also means the only existing lever that trims plugin-EP machinery is `--minimal_build`, which +`static_plugin` currently rejects outright. If measurement shows the size delta matters, lifting that restriction +(already listed as a follow-up) is the mechanism to reach parity — not a new size-reduction flag. + +#### Measured size comparison + +Both modes were then built locally with the exact CI flag set (`Release`, SIMD + threads, asyncify, WebNN on, +`--disable_rtti`, `--enable_wasm_api_exception_catching`, and the full reduced-size set above) and the artifacts +compared: + +| Artifact | `static_lib` | `static_plugin` | Delta | +| --- | ---: | ---: | ---: | +| `ort-wasm-simd-threaded.asyncify.wasm` | 26,550,879 B | 26,488,911 B | **−61,968 B (−0.23%)** | +| `ort-wasm-simd-threaded.asyncify.mjs` | 53,249 B | 53,249 B | 0 | + +(Both binaries rebuilt after merging `main`. An earlier pre-merge pair measured −52,978 B / −0.20%; the merge added +kernels to both builds and did not change the relationship.) + +**There is no size regression** — `static_plugin` is marginally smaller. Both configurations compiled cleanly with +every reduced-size flag, which empirically confirms the analysis above. The plugin build plausibly comes out ahead +because it drops the internal `IExecutionProvider` / `KernelRegistry` glue and the built-in `WebGpuEpFactory` in +`ep_library_internal.cc` in favour of the header-only adapters, but that attribution is not separately measured. +For reference, the same `static_plugin` configuration without any reduced-size flags is 40.9 MB, so those flags are +worth ~15 MB and remain essential regardless of EP mode. + +#### `--enable_wasm_api_exception_catching` and the EP API adapters + +The one option that *did* interact with the plugin path is `--enable_wasm_api_exception_catching`. It is not a +size-reduction flag in the same sense as the others, but it is part of the same CI flag set and it exposed a real +bug. + +`onnxruntime_webassembly.cmake` compiles only `wasm/api.cc` and `core/session/onnxruntime_c_api.cc` with +`-sDISABLE_EXCEPTION_CATCHING=0`; every other translation unit keeps Emscripten's default, where `catch` clauses +are compiled so that they never match. Exceptions still propagate — they are simply only catchable at the C API +boundary. + +The EP API adapters in `include/onnxruntime/ep/adapter/` are header-only and are therefore inlined into WebGPU +translation units that have catching disabled. `OpKernelInfo::GetAttr()` / `GetAttrs()` were implemented as a +`try` around the throwing `Ort::` C++ wrappers, converting `Ort::Exception` into a `Status`. That makes a *missing +optional attribute* — ordinary control flow for `GetAttrOrDefault()` — depend on catching an exception. With +catching disabled the `catch` never ran, so the exception escaped `GetAttrOrDefault()` entirely and surfaced from +session creation as e.g. `ERROR_CODE: 6, ERROR_MESSAGE: No attribute with name:'extrapolation_value'is defined.` + +This reproduced as 949 `suite0` failures on the reduced-size `static_plugin` build while the same build passed on +the CPU backend, the reduced-size `static_lib` build passed, and the full-op `static_plugin` build passed — the +failure required plugin + WebGPU + API-only exception catching together. + +The fix is to not use exceptions for control flow in the adapters: `GetAttr()` / `GetAttrs()` now call the +non-throwing `OrtApi::KernelInfoGetAttribute*` functions directly and convert the returned `OrtStatus*` into a +`Status`. **General rule: header-only EP API adapter code must not rely on catching exceptions**, because a plugin +EP may be compiled with exception catching disabled. The remaining `catch` blocks in `include/onnxruntime/ep` +(`common.h`, `get_capability_utils.h`, `adapter/kernel_registry.h`) sit on genuinely exceptional callback +boundaries rather than on control-flow paths; when catching is disabled those exceptions still reach the outer C +API boundary and are reported, only with a less specific message. + +With that fix the reduced-size `static_plugin` build passes `suite0` on the WebGPU backend in full: 2152 tests, +all passing, matching the `static_lib` control run on the identical flag set. + +#### Measured performance comparison + +The same two binaries were compared on inference performance in Edge, using the test runner's `--perf` mode +(`test-runner-cli model -b=webgpu -e=edge -P=`) against two synthetic models chosen to isolate the two +places the plugin path could plausibly cost something: + +- **dispatch-bound** — 300 chained tiny elementwise ops (`Mul`/`Add`/`Sub`) on a `[1, 1024]` tensor. Almost no GPU + work per node, so the run time is dominated by per-node host-side work, which is exactly what now goes through + the C API adapters. +- **compute-bound** — 16 chained `[512, 512]` `MatMul`s. Host-side per-node cost is negligible against the GPU + work. + +Three runs per model per build. **The two builds must be sampled interleaved** — one round is +`plugin/dispatch, plugin/compute, lib/dispatch, lib/compute`, repeated — rather than all samples of one build +followed by all samples of the other. This machine drifts: a build that has just finished, a warm browser, or a +warm GPU shifts timings by more than the effect being measured, and sequential sampling aliases that drift onto +the build under test. The sequential methodology used for the first pass understated a real regression by nearly +an order of magnitude (see below). The table reports the median of the three per-run P50 values. + +| Metric | `static_lib` | `static_plugin` | Delta | +| --- | ---: | ---: | ---: | +| dispatch-bound, per-run P50 | 11.20 ms | 12.00 ms | +0.8 ms (+7%) | +| compute-bound, per-run P50 | 7.70 ms | 7.80 ms | +0.1 ms (within noise) | + +Reading of these numbers: + +- **Execution is not measurably slower once GPU work dominates.** The compute-bound samples overlap + (`static_plugin` 7.80 / 7.80 / 7.40 against `static_lib` 7.90 / 7.00 / 7.70 — the plugin build's best sample + beats two of the three control samples), so the difference is not resolvable at this sample count. +- **A dispatch-bound graph pays roughly 7%.** That is the cost of routing per-node kernel work through the C API + adapters instead of a direct in-process virtual call. Unlike the compute-bound case this one does separate + cleanly — the plugin build's slowest control sample is still faster than its own fastest sample. It only shows + up when nodes do essentially no GPU work, which is the worst case by construction rather than a representative + one. Amortized over ~300 nodes it is on the order of 3 µs per node. +- **Session creation costs about 3%** (measured pre-merge: 1136 → 1172 ms and 1250 → 1290 ms for the two models). + This covers plugin EP registration, `GetCapability` and kernel creation through the C API. The absolute figure + is dominated by fetching and parsing the model, so the true relative cost of the plugin machinery within session + creation is higher than 3% — but it is tens of milliseconds once, not per inference. + +Two cautions on methodology: + +- The *total* wall time the test runner prints for `suite0` is not a usable metric. It varied between 3 min 15 s + and 4 min 04 s for the **same** `static_lib` binary across two runs. Only the second figure it prints — net test + time, excluding the per-model `before all` session-creation hooks — is stable enough to compare, and by that + measure the two builds are equal. An early single-sample comparison of the total figure suggested a 29% + regression that does not exist. +- Timings taken with `--webgpu.profiling.mode=default` are inflated by the timestamp queries and must never be + compared against timings taken without it. + +#### The shared allocator must defer zero-initialize submission during a run + +Interleaved sampling exposed a genuine, code-level regression in the plugin path that sequential sampling had +hidden: the compute-bound model ran at 9.90 ms against the control's 7.30 ms, **+36%**, with no overlap between +the two sample sets. That the dispatch-bound model — 300 nodes against the compute model's 16 — did *not* show a +proportional cost ruled out a fixed per-node or per-run overhead and pointed at something scaling with bytes +allocated rather than with node count. + +The cause was buffer zero-initialization. `BufferManager::Create` clears a buffer recycled from the cache, and +`GpuBufferAllocator` takes a `should_submit_zero_initialize` predicate deciding whether that clear is submitted +before `Create` returns. During `Session::Run` it must **not** be: the clear is encoded into the run's own command +encoder and rides along with the run's submission, preserving dispatch batching. Outside a run there is no +upcoming submission to piggyback on, so the clear must be submitted immediately. `WebGpuExecutionProvider` and the +plugin factory's per-EP allocator both passed `!IsRunActive()`, but the factory's **shared** allocator — the one +backing intermediate tensors — hardcoded `true`, forcing a queue submit for every intermediate buffer mid-run. +The compute model allocates sixteen 1 MB intermediates, so it paid sixteen forced submits per inference. + +The shared allocator is created by `CreateAllocatorImpl` before any EP instance exists, so it had no EP to ask — +which is why the predicate was stubbed out rather than wired up. The fix moves the run-active flag onto +`WebGpuContext`, which is both reachable from the allocator and the more correct owner: it holds +`current_command_encoder_` / `current_compute_pass_encoder_`, the very state that batching depends on. +`WebGpuExecutionProvider` sets it in `OnRunStart` / `OnRunEnd` and its `IsRunActive()` now delegates to the +context, so all three allocator sites read one source of truth. With the fix the compute-bound gap disappears into +noise, as the table above shows. + +This bug predates the ORT Web migration and affects any consumer of the WebGPU plugin EP, not just the web build; +it is a candidate to land separately from the static-plugin work. + +It was present in the pre-merge measurements too, which reported only +4% for this model. The reason is that the +cost is state-dependent rather than constant: `BufferManager::Create` only clears — and therefore only flushes — +a buffer it recycles from its cache, so the penalty depends on cache occupancy and appears as a wide, roughly +bimodal distribution rather than a fixed offset. Sequential sampling of the *same* defective binary produced +per-run P50s spanning 7.70 ms to 10.20 ms, a range that brackets the pre-merge figure. The merge itself did not +introduce or amplify the defect: of the three merged commits touching this path, #32272 only adds an early return +when an allocator is already registered (not reached with a single device), and #32269's subgroup-matrix MatMul is +gated on the F16 8x16x16 config, so the f32 compute-bound model never takes it. The lesson is about sample count +and interleaving on a high-variance metric, not about the merge. + + + +> Should static factories be registered before environment creation or through environment construction options? + +Neither. They are registered by ORT core during `OrtEnv` creation, after the environment is constructed and +published. See D6. diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md new file mode 100644 index 0000000000000..04309e6f32feb --- /dev/null +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md @@ -0,0 +1,521 @@ +# Static plugin EP registration — work log + +Working notes for the branch `edgchen1/webgpu_static_plugin`. This is a scratch handoff document, not a +design document: the design and its rationale live in +[`static_plugin_ep_registration_design.md`](./static_plugin_ep_registration_design.md), which is the +authoritative reference for decisions D1–D7. **Delete this file before opening the PR.** + +The purpose of this log is to let someone pick the work up on a different machine — in particular a machine +with a real GPU, which the original development machine did not have. + +## Goal + +Make the statically linked WebGPU EP build go through the **plugin EP** path rather than the internal-EP path, +via a generic "static plugin EP" registration mechanism in ORT core. The same EP sources +(`onnxruntime/core/providers/webgpu/ep/*`) then serve both a dynamically loaded plugin DLL and a build that is +statically linked into the host. The latter is what unblocks ORT Web / Emscripten, where shared library modules +are not possible. + +Whether the EP is dynamically or statically **linked** is deliberately separate from whether it is a **plugin**. +Both linkage modes are supported. + +## Status + +The implementation is complete and committed. CI coverage was added for the new configuration, which had none. + +| Area | State | +| --- | --- | +| Implementation | Done | +| Windows build + GPU tests (`--use_webgpu static_plugin`) | Green in CI | +| Shared library plugin build (`--use_webgpu shared_lib`) | Green in CI, no regression from D2/D7 | +| CUDA plugin EP (shares `include/onnxruntime/ep/*`) | Green in CI | +| Operator-level validation on a real GPU | **Done — no behavioural difference**, see [Testing](#testing) | +| Linux/GCC build (`--use_webgpu static_plugin`) | **Green in CI** (run `33030694040`) after the `array-bounds` fix — see [GCC](#gcc-and-warnings-as-errors) | +| Minimal build | Not yet exercised — see [Open items](#open-items) | +| Emscripten / ORT Web | **Builds and links**, plugin registration verified in the binary — see [ORT Web](#emscripten-and-ort-web). Not yet run in a browser | +| PR | Not opened yet | + +## Branch + +- Branch: `edgchen1/webgpu_static_plugin`, pushed to `git@github.com:microsoft/onnxruntime.git` (not a fork — + this matters, because the self-hosted 1ES CI pools are not available to forks). +- Forked from `main` at `fab4691797560a22092d9bbe8cec540b5413724b`. +- The earlier commits on the branch (`dff9cc42bf` .. `800ad881ef`) are the WebGPU extraction planning documents + and are unrelated to this workstream's code. The code work starts at `2754674229`. + +| Commit | Summary | +| --- | --- | +| `2754674229` | Support registering statically linked plugin EPs, and build WebGPU EP that way | +| `32775c305a` | Add CI jobs for the WebGPU static plugin EP build | +| `49855f6ef7` | Fix GCC maybe-uninitialized errors in the EP adapter attribute accessors | +| `d47f00a0a6` | Update dynamic plugin EP test config parsing tests | +| `1980e901ec` | Record CI coverage in the static plugin EP registration design doc | +| `754e025ddc` | Add a work log for the static plugin EP branch | +| `a362107df9` | Merge `origin/main` (52 commits behind at the time; merged, not rebased) | +| `35e6f65d52` | Record real-GPU validation results in the work log | +| `a110948b1d` | Record GPU validation and the plugin-path test coverage gap in the design doc | +| `8d87c11972` | Avoid GCC 14 `-Warray-bounds` false positive in `EmitGateActivationExpr` — the batch-3 CI fix | +| `ef68884a95` | Record the complete GCC 14 warning inventory and the local reproduction recipe | + +Pushed through `ef68884a95`. The push from `754e025ddc` was a fast-forward; no history has been rewritten on +this branch. + +## What the change consists of + +Read the design doc first. The short version, keyed to the decisions: + +- **D1** — A link-time registry in ORT core (`onnxruntime/core/session/plugin_ep/ep_static_plugins.{h,cc}`), + hand-written and `#if`-guarded, mirroring `EpLibraryInternal::CreateInternalEps`. Code generation from CMake + and a new public C API were both considered and rejected. +- **D2** — `ORT_PLUGIN_EP_ENTRY_POINT_PREFIX` in `onnxruntime/core/providers/webgpu/ep/api.cc`. The shared + library build must keep exporting **unprefixed** `CreateEpFactories` / `ReleaseEpFactory`, because + `EpLibraryPlugin::Load` looks those up by name. The static build emits `WebGpu_`-prefixed symbols instead. +- **D3** — No new ABI hook for process-global teardown. `ShutdownProtobufLibrary()` is compiled out unless + `ORT_PLUGIN_EP_OWNS_PROCESS_GLOBALS`. The legacy WebGPU cleanup block in `ort_env.cc` must stay for the + default internal-EP build. +- **D4** — CMake option `onnxruntime_WEBGPU_STATIC_PLUGIN` plus the derived + `onnxruntime_WEBGPU_LINKED_INTO_HOST`; surfaced as `--use_webgpu static_plugin`. +- **D5** — `ep_library_path` became optional in the dynamic plugin EP test infrastructure's + `InitializationConfig`, so a statically registered EP can be selected without a path to load. +- **D6** — Publish-then-register in `ort_env.cc`, and `OrtEnv::m_` became a `std::recursive_mutex`. Registration + reenters the environment (`GetSupportedDevices()` → `Api().ep.GetEnvConfigEntries()` → + `OrtEnv::TryGetInstance()`), which self-deadlocked with a plain mutex. +- **D7** — `ORT_PLUGIN_EP_STATICALLY_LINKED` disables the manual C++ API init in the shared + `include/onnxruntime/ep/api.h`. Do not try to force `ORT_API_MANUAL_INIT` on for everyone: MSVC's + `#pragma detect_mismatch` requires whole-binary agreement and doing so produced roughly 50 `LNK2038` errors. + +Files touched outside `docs/`: + +``` +.github/workflows/linux_webgpu.yml +.github/workflows/windows_webgpu.yml +cmake/CMakeLists.txt +cmake/onnxruntime.cmake +cmake/onnxruntime_providers_webgpu.cmake +cmake/onnxruntime_python.cmake +cmake/onnxruntime_unittests.cmake +include/onnxruntime/core/session/environment.h +include/onnxruntime/ep/adapter/op_kernel_info.h +include/onnxruntime/ep/api.h +onnxruntime/core/providers/webgpu/ep/api.cc +onnxruntime/core/providers/webgpu/math/{gemm,softmax,top_k}.h +onnxruntime/core/providers/webgpu/tensor/cast.h +onnxruntime/core/session/environment.cc +onnxruntime/core/session/ort_env.{h,cc} +onnxruntime/core/session/plugin_ep/ep_library_plugin.cc +onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.{h,cc} (new) +onnxruntime/core/session/plugin_ep/ep_library_static_plugin.{h,cc} (new) +onnxruntime/core/session/plugin_ep/ep_static_plugins.{h,cc} (new) +onnxruntime/test/framework/dynamic_plugin_ep_test.cc +onnxruntime/test/framework/inference_session_test.cc +onnxruntime/test/unittest_main/test_main.cc +onnxruntime/test/unittest_util/test_dynamic_plugin_ep.{h,cc} +tools/ci_build/{build.py,build_args.py} +``` + +## Building + +Three WebGPU configurations exist and it is worth being able to build at least two of them, because several +bugs in this work only appeared in one of them: + +``` +python tools/ci_build/build.py ... --use_webgpu # internal EP (the default, pre-existing path) +python tools/ci_build/build.py ... --use_webgpu shared_lib # plugin EP in its own shared library +python tools/ci_build/build.py ... --use_webgpu static_plugin # plugin EP linked into the host <-- the new one +``` + +`--use_webgpu static_plugin` maps to `-Donnxruntime_USE_EP_API_ADAPTERS=ON +-Donnxruntime_WEBGPU_STATIC_PLUGIN=ON` (see `tools/ci_build/build.py`, the `--use_webgpu` handling). Both +options default to `OFF`. + +`onnxruntime_USE_EP_API_ADAPTERS=ON` is **incompatible** with `onnxruntime_BUILD_DAWN_SHARED_LIBRARY=ON` +(`cmake/CMakeLists.txt` raises a `FATAL_ERROR`). The internal-EP WebGPU CI job sets the latter; the plugin jobs +must not. + +### Windows gotchas + +These cost real time on the original machine and are likely to recur: + +- **`MAX_PATH`.** Building WebGPU under a long path such as `\build\Windows` makes Dawn's DXC build fail + with `MSB3491`. Build into a short directory instead — `C:\b` and `C:\b2` were used for the `static_plugin` + and `shared_lib` configurations respectively. With the Visual Studio generator the binaries land in + `\\\`. + + Refinement from a later machine: this is really an **MSBuild** problem. With `--cmake_generator Ninja` and + the registry's `LongPathsEnabled=1`, building at `D:\source\onnxruntime_4\build\sp` worked fine, so an + in-repo build directory is usable if you avoid MSBuild. Note that with Ninja the binaries land in + `\\`, one level, not two. +- **CMake may not be on `PATH`.** On a Visual Studio install it is at + `C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin`. +- Incremental rebuild of just the EP is much faster than a full build: + `cmake --build --config Debug --target onnxruntime_providers_webgpu --parallel`. + +### Verifying the D2 export contract + +For the **shared library** configuration, confirm the exported entry points are still unprefixed, since +`EpLibraryPlugin::Load` resolves them by name: + +``` +dumpbin /exports \\\onnxruntime_providers_webgpu.dll +``` + +Expect exactly `CreateEpFactories` and `ReleaseEpFactory`. The `static_plugin` build should instead produce +`WebGpu_`-prefixed symbols and no DLL. + +## Testing + +Relevant tests: + +- `onnxruntime_test_all --gtest_filter=InferenceSessionTests.WebGpu*` — covers registration, device + enumeration, and session creation end to end through the public V2 API. +- `onnxruntime_provider_test` — WebGPU EP kernel tests. + +The original development machine was a **Hyper-V VM with no real GPU**. On such a host WebGPU only surfaces a +*virtual* `OrtEpDevice`, and any session that is not compile-only is rejected up front with +*"WebGPU EP was selected on a virtual GPU device…"*. So locally only these two pass: + +- `InferenceSessionTests.WebGpuVirtualDeviceCompileOnlyEndToEnd` +- `InferenceSessionTests.WebGpuVirtualDeviceRejectedWithoutCompileOnly` + +and `WebGpuCompileOnlySkipsFinalization` and `TestStrictShapeInference` fail. This was proven to be a property +of the host and not of static linking, by building `--use_webgpu shared_lib` on the same machine and observing +the identical pass/fail set. + +**Consequence for whoever picks this up:** operator execution cannot be validated on a GPU-less machine at all. +Two ways to get real coverage: + +1. **Use CI.** The new `webgpu_static_plugin_build_x64_RelWithDebInfo` job in `.github/workflows/windows_webgpu.yml` + runs on the `onnxruntime-github-Win2022-GPU-A10` pool, which is the only GPU-equipped WebGPU pool, and it + both builds and tests. It has passed. This is currently the primary source of operator-level truth. +2. **Run locally on a real GPU.** Nothing in the change is GPU-vendor specific; a normal Windows machine with a + D3D12 or Vulkan capable GPU is enough. + +### Results on a real GPU + +This has now been done, on a machine with an NVIDIA RTX 5060 Ti. Both configurations were built from the same +tree, RelWithDebInfo, Ninja, differing **only** in the EP path: + +``` +python tools\ci_build\build.py --config RelWithDebInfo --build_dir --skip_submodule_sync --parallel \ + --cmake_generator Ninja --use_webgpu [static_plugin] --update --build --skip_tests \ + --cmake_extra_defines onnxruntime_BUILD_UNIT_TESTS=ON \ + onnxruntime_ENABLE_DAWN_BACKEND_D3D12=1 onnxruntime_ENABLE_DAWN_BACKEND_VULKAN=1 +``` + +`onnxruntime_provider_test`, with `ONNXRUNTIME_TEST_GPU_DEVICE_ID=0`: + +| | static_plugin | internal EP | +| --- | --- | --- | +| Total | 5894 | 5938 | +| Passed | 5775 | 5803 | +| Failed | 0 | 0 | +| Skipped | 119 | 135 | + +**Across the 5894 tests common to both builds there were zero status differences.** This is the evidence that +routing the static build through the plugin path changes no operator behaviour. + +`onnxruntime_test_all`: 1965 passed on both, no differences. All three `InferenceSessionTests.WebGpu*` pass on +the `static_plugin` build, including `WebGpuCompileOnlySkipsFinalization` — confirming the earlier failure of +that test was a property of the GPU-less host, as suspected. + +Two categories of test legitimately differ between the builds; neither is a regression: + +- **44 tests exist only in the internal-EP build.** 16 are `DISABLED_` and never run. The other 28 are + white-box tests that include internal WebGPU EP headers (`WebGpuContextTest`, `ActivationCacheKeyTest`, + `MatMul2BitsWebGpu`, `HardSwish_WebGPU`); `cmake/onnxruntime_unittests.cmake` (the + `onnxruntime_USE_WEBGPU AND NOT onnxruntime_USE_EP_API_ADAPTERS` guard) excludes `test/providers/webgpu/*` + from adapter builds entirely. Of these, `HardSwish_WebGPU` and `MatMul2BitsWebGpu` (12 tests) are real + operator coverage that exists only on the internal path — a known coverage gap of the plugin path, worth + either documenting in the design doc or closing by porting them to EP-agnostic tests. +- **The `InferenceSessionTests.WebGpu*` virtual-device tests are deliberately paired, not shared.** The + internal build has `WebGpuEpFactoryVirtualDevice`, `WebGpuEpFactoryRejectsVirtualDeviceWithoutCompileOnly` + and `WebGpuCompileOnlyUsesNoOpAllocator`; the plugin build has `WebGpuVirtualDeviceCompileOnlyEndToEnd` and + `WebGpuVirtualDeviceRejectedWithoutCompileOnly`, which `inference_session_test.cc` documents in-source as the + plugin-build counterparts. Only `WebGpuCompileOnlyUsesNoOpAllocator` has no counterpart, because it asserts + an internal allocator *type* via `dynamic_cast`, which is inherently unobservable across the ABI boundary. + +One caveat for whoever repeats this: do **not** run the two `onnxruntime_test_all` binaries concurrently. +`PathValidationTest`'s symlink tests collide over a shared temp directory and produce a spurious, and +non-deterministic, mix of pass/fail/skip. Run sequentially: all 31 pass on both builds. + +There is also a `webgpu-local-testing` skill in `.agents/skills/` describing how to run WebGPU tests on Linux +without a GPU using Mesa lavapipe (software Vulkan). Note its stated limitation: any graph containing `MatMul` +crashes lavapipe, so it only validates host-side enforcement and shape logic. + +### Not covered by any test yet + +Re-registration — register, run, unregister, re-register — is untested and is not reachable from the existing +test binaries, because `OrtEnv` is a refcounted process singleton that the unit test main pins for the lifetime +of the process. + +## CI + +No pre-existing leg built the `static_plugin` configuration, since both CMake options default to `OFF`. Every +WebGPU job was either internal EP (`--use_webgpu`) or shared library plugin (`--use_webgpu shared_lib`). Two +jobs were added: + +- `webgpu_static_plugin_build_x64_RelWithDebInfo` in `.github/workflows/windows_webgpu.yml` — builds **and + tests** on the A10 GPU pool. It deliberately omits `onnxruntime_BUILD_DAWN_SHARED_LIBRARY=ON` (incompatible, + see above) and deliberately drops the `--disable_rtti` / `--enable_lto` flags that the shared library job + carries, since those describe how the plugin ships as a DLL and here they would apply to all of ORT. +- `build-linux-webgpu-static-plugin-x64-release` in `.github/workflows/linux_webgpu.yml` — build only, mirroring + its sibling job. This is the only **GCC** coverage of the EP adapters, and the only coverage of the + `cmake/onnxruntime_python.cmake` change, which is why `--build_wheel` is kept on it. + +`linux_webgpu.yml` is build-only by design; its test job is commented out with a pre-existing "currently +failing" TODO. + +### Running CI on this branch + +The workflows only trigger automatically on `push`/`pull_request` for `main`, `rel-*`, and +`plugin-ep-webgpu/rel-*`, so **pushing this branch triggers nothing**. Dispatch manually: + +``` +gh workflow run windows_webgpu.yml --repo microsoft/onnxruntime --ref edgchen1/webgpu_static_plugin +gh workflow run linux_webgpu.yml --repo microsoft/onnxruntime --ref edgchen1/webgpu_static_plugin +gh workflow run windows_cuda_plugin.yml --repo microsoft/onnxruntime --ref edgchen1/webgpu_static_plugin +``` + +`gh workflow run --ref ` reads the workflow file **from that ref**, so jobs added on the branch do run +even though they are not on `main` yet. This was confirmed empirically. + +`windows_cuda_plugin.yml` matters because the CUDA plugin EP shares `include/onnxruntime/ep/*` with WebGPU, so +it is the regression check for the D5 and D7 changes. + +### Retrieving logs + +`gh run view --log-failed` and `gh run view --job --log` both returned **zero bytes** for these runs. Use +the REST API instead: + +``` +gh api repos/microsoft/onnxruntime/actions/runs//jobs --jq '.jobs[] | select(.conclusion=="failure") | "\(.id) \(.name)"' +gh api repos/microsoft/onnxruntime/actions/jobs//logs > log.txt +``` + +When grepping the result, search for `FAILED:` **case-sensitively**. A case-insensitive search for "failed" or +a search for `error:` drowns in benign CMake and Dawn feature-probe output such as +`-- Performing Test ... - Failed`. The same trap applies to local build logs: `grep -c "error:"` matches Dawn's +own configure line `-- DAWN Werror: OFF`, which looks alarming in a summary count and is nothing. + +### Run history + +| Run | Workflow | Result | +| --- | --- | --- | +| 32914301463 | ONNX Runtime WebGPU Builds | Success, all 6 jobs including the new one | +| 32914304408 | Linux WebGPU CI | Failure — GCC `maybe-uninitialized`, batch 1 | +| 32914307098 | CUDA Plugin Windows CI | Failure — `ParseInitializationConfigRejectsMissingRequiredFields` | +| 32996643911 | Linux WebGPU CI | Failure — GCC `maybe-uninitialized`, batch 2 | +| 32996646394 | CUDA Plugin Windows CI | Success | +| 33009901327 | Linux WebGPU CI | Failure — GCC `array-bounds`, batch 3. Internal-EP job on the same run passed. | +| 33030694040 | Linux WebGPU CI | **Success** — both jobs. Confirms the `array-bounds` fix; first green GCC run on this branch. | +| 33030695742 | ONNX Runtime WebGPU Builds | Dispatched on `ef68884a95` — re-run after the merge from `main` | +| 33030697208 | CUDA Plugin Windows CI | Dispatched on `ef68884a95` — re-run after the merge from `main` | + +## GCC and warnings as errors + +This was the only substantial problem the new CI surfaced, and it will bite again, so it is worth understanding +rather than pattern-matching. + +The EP adapter types are defined **entirely inline in headers**. The in-tree `OpKernelInfo` hides its +accessors behind out-of-line definitions in a `.cc`, so GCC cannot see their failure and exception paths. With +the adapters, it can — and it then warns about values that are only ever read once the call has succeeded. The +warning is a false positive, but the build uses `-Werror`. + +This had never been observed before simply because the EP-adapters path had never been compiled with GCC: the +existing Linux WebGPU CI builds the internal EP, and the shared library plugin builds are Windows/MSVC-only. +The CUDA plugin EP hit the identical problem earlier and solved it with a target-scoped suppression in +`cmake/onnxruntime_providers_cuda_plugin.cmake`. + +Two batches of failures appeared, because ninja stops early and only reports the targets that were already in +flight — **do not assume the first batch is the complete list**. A third batch, of a different warning class, +appeared later still. The lesson is the general one: **each CI round trip reveals only one batch**, so prefer a +local GCC build (or a throwaway dispatch with `--compile_no_warning_as_error`) to enumerate all of them at once. + +- Batch 1, fixed at the source in `49855f6ef7`: uninitialized locals that are passed by pointer and read back, + and the temporaries inside `GetAttrOrDefault` / `GetAttr` in `include/onnxruntime/ep/adapter/op_kernel_info.h`. + Value-initializing them (`T tmp{}`) is correct and worth keeping on its own merits. +- Batch 2: false positives deep inside `absl::InlinedVector`, for `TensorShapeVector` locals in + `tensor/pad.cc` and `tensor/unsqueeze.h` that are always properly constructed. These cannot be fixed at the + source. Handled by applying the same `-Wno-maybe-uninitialized` suppression the CUDA plugin EP uses, scoped + to the `onnxruntime_providers_webgpu` target, GCC only, and only when `onnxruntime_USE_EP_API_ADAPTERS` is on + so that non-adapter builds stay strict. +- Batch 3, in run `33009901327`, **fixed at the source**: `-Werror=array-bounds`, a *different* warning class, at + `onnxruntime/contrib_ops/webgpu/quantization/matmul_nbits_mlp.cc:47` in `EmitGateActivationExpr`: + + ``` + ‘void* __builtin_memcpy(...)’ forming offset [32, 48] is out of the bounds [0, 32] + of object ‘’ with type ‘std::__cxx11::basic_string’ + ``` + + GCC 14 at `-O3`, inlining the `std::string{gate_var} + " * (one / (one + exp(-" + std::string{gate_var} + ")))"` + temporary-move concat chain through `std::operator+(basic_string&&, const char*)`. A false positive. Note that + the batch-2 `-Wno-maybe-uninitialized` suppression **is** working — it is present on the failing command line + and the batch-2 files compiled clean. The file is pre-existing (added by PR #28280, `88ca23fcd5`), not authored + by this branch; the adapters PCH pulls in `include/onnxruntime/ep/adapters.h`, which changes inlining and is + why only the adapters config trips it. Fixed by building the expression with ORT's `MakeString` instead of a + `std::string` concat chain — a single-site source fix, preferred over extending the target-scoped suppression + with `-Wno-array-bounds`, which would have blinded the whole provider to a genuinely useful warning. + +### The complete GCC 14 warning inventory + +The batch-at-a-time discovery loop was finally closed by enumerating **every** warning in one pass: a full +container build with `--compile_no_warning_as_error`, then grouping the log by warning class. The result: + +| Warning class | Count | Where | +| --- | --- | --- | +| `-Wattributes` | 276 | Dawn generated code (`dawn_proc.cpp`) | +| `-Wunused-parameter` | 55 | ONNX (`onnx/defs/**`) | +| `-Wdangling-pointer=` | 3 | Dawn / Tint | +| `-Wchanges-meaning` | 3 | Dawn (`SystemEvent.h`) | +| `-Wstringop-overflow=` | 2 | flatbuffers (`reflection.cpp`) | +| `-Warray-bounds=` | **1** | **ORT — `matmul_nbits_mlp.cc:47`** | +| `-Wstringop-overread` | 1 | telemetry (`sqlite3_retail.c`) | + +**Exactly one warning in the whole build is attributable to ORT's own source**, and it is the batch-3 failure. +Everything else is third-party, in targets that do not compile with `-Werror`. So there is no batch 4: fixing +this one site is expected to turn the leg green. + +### Reproducing the CI compiler locally + +Worth the setup cost — it turns a ~28 minute CI round trip per batch into a local loop, and it is the only way +to enumerate all batches at once. + +The CI base image (`onnxruntimebuildcache.azurecr.io/.../cpu_x64_almalinux8_gcc14`) requires ACR auth and is +**not pullable**. An equivalent image is easy to build, and reproduces the compiler exactly +(GCC 14.2.1 20250110, Red Hat) — `almalinux:8` + `epel-release` + `gcc-toolset-14` + `python3.11`, with +`cmake ninja packaging numpy setuptools wheel` from pip. Notes: + +- **Python 3.11 is required, not the distro's 3.9** — `tools/ci_build/build_args.py` uses a `match` statement + (3.10+). CI itself uses `/opt/python/cp310-cp310/bin`. +- `--allow_running_as_root` is required inside the container. +- **No X11 dev packages are needed**, because `cmake/external/onnxruntime_external_deps.cmake` sets + `DAWN_USE_X11 OFF` unconditionally. This is what keeps the image minimal. +- Clone into the WSL native filesystem, not `/mnt/d` — building over the 9p mount is crippling. Only + `cmake/external/onnx` and `cmake/external/libprotobuf-mutator` need initialising for a native Linux build. + +To re-check a single file without paying for a whole build, override `CXX_FLAGS` on the generated makefile +rule, which lets you promote warnings back to errors against an existing build tree: + +```bash +FLAGS=$(sed -n 's/^CXX_FLAGS = //p' CMakeFiles/onnxruntime_providers_webgpu.dir/flags.make) +make -f CMakeFiles/onnxruntime_providers_webgpu.dir/build.make \ + CMakeFiles/onnxruntime_providers_webgpu.dir/src//matmul_nbits_mlp.cc.o \ + CXX_FLAGS="$FLAGS -Werror" +``` + +Always run the **negative control** — rebuild the unfixed source through the same harness and confirm it still +errors. Otherwise a harness that silently fails to exercise the warning looks exactly like a successful fix. + +### Verification of the fix + +Both halves were checked, because each one alone is inconclusive: + +1. **Single-object recheck.** With the fix, `matmul_nbits_mlp.cc.o` compiles clean under `-Werror`. Reverting the + file and rebuilding through the identical harness reproduces the exact CI error, which is what proves the + harness was actually exercising the warning. +2. **Full build with `-Werror` on**, i.e. the real CI configuration with `--compile_no_warning_as_error` dropped: + completes to 100%, links `onnxruntime_provider_test`, zero compiler errors. Before trusting this, confirm + `-Werror` really is on the ORT targets — `sed -n 's/^CXX_FLAGS = //p' CMakeFiles/.dir/flags.make` + should show it for `onnxruntime_providers_webgpu`, `onnxruntime_providers` and `onnxruntime_session` — and + confirm the fixed file appears in the build log, so it was recompiled rather than served from a stale object. + +The 334 warnings that remain in that build are all third-party, in targets that do not use `-Werror`. + +## Emscripten and ORT Web + +This is the motivating use case for the whole change, so it was built locally before opening the PR. **It builds +and links, and the plugin registration is verifiably present in the output.** + +Note what ORT Web's own CI does *not* cover today: the WebGPU WASM jobs in +`.github/workflows/linux-wasm-ci-build-and-test-workflow.yml` pass a bare `--use_webgpu`, and bare `--use_webgpu` +defaults to `static_lib` (`tools/ci_build/build_args.py`, `const="static_lib"`). So every existing WASM job still +builds the **internal** EP. Nothing in CI exercises WASM + `static_plugin`; this branch only makes the +configuration available, it does not switch ORT Web over to it. + +The CMake side was already written with Emscripten in mind — `cmake/onnxruntime_providers_webgpu.cmake` rejects +`shared_lib` there with `"WebGPU EP shared library build is not supported on Emscripten. Please use +'--use_webgpu static_plugin'."` + +### Building it + +Same container as the GCC 14 reproduction (see [above](#reproducing-the-ci-compiler-locally)), plus: + +- `git submodule update --init --recursive cmake/external/emsdk` — not needed for a native Linux build, required + here. `build.py` installs and activates the toolchain itself. +- **`node` must be on `PATH`.** `cmake/external/onnxruntime_webassembly.cmake` includes `node_helper.cmake`, + which does a hard `find_program(NODE_EXECUTABLE ...)` and fails configure with + `Could not find NODE_EXECUTABLE using the following names: node.exe, node`. CI gets node from + `actions/setup-node`; the AlmaLinux image has none. emsdk bundles one — point `PATH` at + `cmake/external/emsdk/node//bin`. + +``` +docker run --rm -v /home/edch/ort:/src -v /home/edch/ortwasm:/build -w /src ort-gcc14:local \ + bash -lc "export PATH=/src/cmake/external/emsdk/node/22.16.0_64bit/bin:\$PATH && \ + python3.11 tools/ci_build/build.py --build_dir /build --config Release --skip_submodule_sync --parallel \ + --allow_running_as_root --build_wasm --enable_wasm_simd --enable_wasm_threads \ + --enable_wasm_api_exception_catching --use_webgpu static_plugin --use_webnn \ + --target onnxruntime_webassembly --skip_tests" +``` + +Result: configure clean, zero compile failures, links `ort-wasm-simd-threaded.asyncify.mjs` + +`.wasm` (~32 MB). Roughly 35 minutes including the emsdk toolchain download and sysroot generation. + +### Verifying the plugin path actually linked + +A successful build alone proves little here, because the registration is behind +`#if defined(USE_WEBGPU) && defined(ORT_WEBGPU_STATIC_PLUGIN)` in +`onnxruntime/core/session/plugin_ep/ep_static_plugins.cc`. If that define failed to reach ORT core, the file +would still compile — `CreateStaticPluginEpLibraries()` would just return an empty vector, and the build would +be just as green while registering nothing. + +The decisive check is that ORT core holds an **undefined** reference to the prefixed entry points, using the +emsdk LLVM tools: + +``` +NM=cmake/external/emsdk/upstream/bin/llvm-nm +$NM /Release/libonnxruntime_session.a | grep WebGpu_ # expect: U WebGpu_CreateEpFactories, U WebGpu_ReleaseEpFactory +$NM /Release/libonnxruntime_providers_webgpu.a | grep WebGpu_ # expect: T WebGpu_CreateEpFactories, T WebGpu_ReleaseEpFactory +``` + +Observed exactly that: `U` in core, `T` in the provider, and the link resolved them. That closes the loop — +the `#if` was live, the entry point prefixing worked, and `EpLibraryStaticPlugin` is in the binary. + +### Still open + +The binary has not been **run**. `--skip_tests` was used, and exercising it needs a browser with WebGPU, which +the container does not have. So this validates build, link and registration wiring on Emscripten, not runtime +behaviour. + +## Open items + +1. ~~**Fix the Linux leg.**~~ **Done.** The batch-3 `array-bounds` break is fixed at the source, and a local + GCC 14 build confirmed it was the *only* ORT-owned warning in the whole build (see + [The complete GCC 14 warning inventory](#the-complete-gcc-14-warning-inventory)). CI run `33030694040` + is green on both jobs. +2. ~~**Operator-level validation on a real GPU.**~~ **Done** — see [Results on a real GPU](#results-on-a-real-gpu). + No behavioural difference. Remaining sub-item: decide whether to close the 12-test + `HardSwish_WebGPU` / `MatMul2BitsWebGpu` coverage gap on the plugin path, or document it as known. +3. ~~**Minimal build.**~~ **Resolved.** Exactly one pipeline combines an extended-minimal build with WebGPU — + `webgpu_minimal_build_edge_build_x64_RelWithDebInfo` in `.github/workflows/windows_webgpu.yml`, which passes a + bare `--use_webgpu` (internal EP) and is green, so the `#if !defined(ORT_MINIMAL_BUILD)` guard in `ort_env.cc` + is already exercised in CI. No ORT Web or wasm job uses a minimal build at all. The remaining hole — minimal + plus `static_plugin` linking the EP in without registering it — is now a configure-time `FATAL_ERROR` in + `cmake/CMakeLists.txt`, verified to fire on that combination and not on a non-minimal `static_plugin` + configure. Lifting the restriction properly is follow-up work, recorded in D4 of the design doc. +4. ~~**Emscripten / ORT Web.**~~ **Builds and links**, with the registration verified in the binary — see + [Emscripten and ORT Web](#emscripten-and-ort-web). Not yet run in a browser. +5. **Follow-up cleanup** listed at the end of the design doc, including removing the WebGPU special cases that + only exist while it is still an "internal" EP. +6. **Open the PR.** + +## Environment notes + +Specific to the original machine, but the class of problem generalises. + +- The working copy `C:\source\ort-webgpu-static-plugin` is a **linked git worktree** of `C:\source\onnxruntime`. + `.git` is therefore a *file*, not a directory. Do not write scratch files into it. +- Linting is `lintrunner`, driven by the `ort-lint` skill. `lintrunner -a` can report "ok No lint issues" and + "Successfully applied all patches" in the same run, so always re-check `git status` afterwards. +- The repository's own pre-commit hook lives in `.githooks/pre-commit` (this is not the `pre-commit` framework) + and is enabled with `git config core.hooksPath .githooks`. It **silently does nothing if `lintrunner` is not + on `PATH`**, so activate the virtual environment before committing or the commit goes through unlinted. diff --git a/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md b/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md index a704ea097a5c3..e98952daeb12b 100644 --- a/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md +++ b/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md @@ -55,6 +55,9 @@ The effort is divided into four workstreams: | `test-conformance` | [Test ownership and operator conformance](test_ownership_and_conformance/test_ownership_and_conformance_workstream.md) | Existing coverage is preserved, every test has an owner, and portable conformance coverage protects the external provider | | `node-migration` | [Node plugin migration](node_plugin_migration/node_plugin_migration_workstream.md) | Existing bundled Node WebGPU support is replaced by an explicitly consumable plugin without regressing current users | +The detailed static plugin EP registration design is in +[Static Plugin EP Registration](plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md). + The detailed reusable conformance-suite design is in [Execution Provider Operator Conformance Suite](test_ownership_and_conformance/ep_operator_conformance_design.md). From fdf2f53e0a3d516d5aa20454ceb86c3f649a043b Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Tue, 1 Sep 2026 21:19:07 -0700 Subject: [PATCH 034/107] docs: tighten WebGPU EP extraction plan docs Remove the static plugin EP registration design doc and its work log, which belong to an in-progress branch, along with the references they added to the overview and the plugin-boundary workstream. Give each fact, gate, and open question a single owner. The overview drops the delivery stages, the cutover list, and the cross-workstream open-questions list; those steps and questions now live in the workstream that owns them. Conformance terminology moves to the test-conformance workstream, and the cross-repository version policy is promoted to a top-level section in provider-isolation. Add a size and latency baselines work package to plugin-boundary, whose completion criteria referenced baselines that no work package established. Split the conformance dataset open question into versioning, deduplication, and distribution. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- ...boundary_and_web_integration_workstream.md | 23 +- .../static_plugin_ep_registration_design.md | 717 ------------------ .../static_plugin_ep_work_log.md | 521 ------------- ...ion_and_repository_migration_workstream.md | 5 +- .../ep_operator_conformance_design.md | 6 +- ...st_ownership_and_conformance_workstream.md | 11 +- .../webgpu_ep_extraction.md | 113 +-- 7 files changed, 55 insertions(+), 1341 deletions(-) delete mode 100644 docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md delete mode 100644 docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md index 8dcbba70d3615..f243fbc9c7da4 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md @@ -46,8 +46,6 @@ The design must address: The facility must be generic and validated with at least one non-WebGPU test plugin where practical. -The detailed design is in [Static Plugin EP Registration](static_plugin_ep_registration_design.md). - ## Process-global ownership and teardown Static registration removes the dynamic-library unload boundary. The current WebGPU plugin cleanup path cannot be @@ -138,14 +136,17 @@ The bridge should not expose unrelated ORT private implementation details. ## Work packages -1. **Static registration core:** implement and contract-test generic static factory registration. -2. **Global lifetime contract:** inventory process-global state and implement safe ownership and teardown rules. -3. **Emscripten prototype:** compile the adapter path statically and run a small model. -4. **Gap inventory triage:** convert private-dependency findings into public API or provider-owned actions. -5. **Browser bridge:** specify and prototype object and lifetime exchange. -6. **Parity and retirement:** run the existing suite through the plugin path and remove the legacy path. +1. **Size and latency baselines:** measure `onnxruntime-web` WebAssembly size and inference latency before any + plugin-path change lands, since the size and latency completion criteria are relative to them. +2. **Static registration core:** implement and contract-test generic static factory registration. +3. **Global lifetime contract:** inventory process-global state and implement safe ownership and teardown rules. +4. **Emscripten prototype:** compile the adapter path statically and run a small model. +5. **Gap inventory triage:** convert private-dependency findings into public API or provider-owned actions. +6. **Browser bridge:** specify and prototype object and lifetime exchange. +7. **Parity and retirement:** run the existing suite through the plugin path and remove the legacy path. -The first five packages can proceed largely in parallel. Legacy-path removal waits for their convergence. +Baselines come first, because they cannot be captured once the plugin path has changed anything. Packages 2 through 6 +can then proceed largely in parallel. Legacy-path removal waits for their convergence. ## Interfaces with other workstreams @@ -178,6 +179,6 @@ The first five packages can proceed largely in parallel. Legacy-path removal wai ## Open questions -- Resolved: neither. ORT core registers static factories during `OrtEnv` creation, after the environment is - constructed and published. See [Static Plugin EP Registration](static_plugin_ep_registration_design.md). +- Should static factories be registered before environment creation or through environment construction options? - What is the stable representation of JavaScript-owned WebGPU objects at the C API boundary? +- Which private-dependency findings require a public plugin EP API addition rather than a provider-owned replacement? diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md deleted file mode 100644 index 41e2f6ed0b062..0000000000000 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ /dev/null @@ -1,717 +0,0 @@ -# Static Plugin EP Registration - -Status: Detailed design supporting -[Plugin Boundary and Web/Wasm Integration](plugin_boundary_and_web_integration_workstream.md). - -## Purpose - -Define how a plugin execution provider (EP) that is linked into the host binary is registered with ONNX Runtime, so -that statically linked and dynamically loaded plugin EPs share one provider implementation and one public API -boundary. - -WebGPU is the first consumer. The immediate deliverable is to move the statically linked WebGPU build off -`EpLibraryInternal` and onto the plugin EP path. - -## Background - -ORT has two `EpLibrary` implementations today: - -- `EpLibraryPlugin` (`onnxruntime/core/session/plugin_ep/ep_library_plugin.cc`) loads a shared library, resolves the - `CreateEpFactories` and `ReleaseEpFactory` symbols, and drives the factory lifecycle. -- `EpLibraryInternal` (`onnxruntime/core/session/plugin_ep/ep_library_internal.cc`) wraps an in-tree - `IExecutionProvider`, including `EpLibraryInternal::CreateWebGpuEp`. - -There is no `EpLibrary` that accepts factory entry points directly, so a provider compiled into the host cannot reach -the plugin path. - -This blocks ORT Web specifically. `cmake/onnxruntime_providers_webgpu.cmake` raises a `FATAL_ERROR` for the WebGPU -shared-module build under Emscripten, so static linking is the only way a Wasm build can ever reach the plugin -boundary. - -## Goals - -- Register a plugin EP that is linked into the host, reusing the existing plugin registration path after the point - where symbol lookup would occur. -- Present statically linked plugin code with the same runtime conditions as dynamically loaded plugin code. -- Require no change to provider sources between the two linkage modes. -- Avoid new host call sites for in-tree providers. -- Keep the shared-library path byte-for-byte unchanged at runtime. - -## Non-goals - -- A public C API for registering static plugin EPs. Deferred until a provider is genuinely out-of-tree. -- Removal of the EP API adapters. Tracked separately. -- Removal of the direct `IExecutionProvider` WebGPU path. That is the final step of the workstream, not this change. -- Support for static plugin EPs in minimal builds. `cmake/onnxruntime_session.cmake` excludes all of - `core/session/plugin_ep/` when `onnxruntime_MINIMAL_BUILD` is set. - -## Decisions - -### D1: Registration mechanism is a link-time registry in ORT core - -ORT core holds a hand-written, `#if`-guarded list of statically linked plugin EP entry points, mirroring the shape of -`EpLibraryInternal::CreateInternalEps`. A new `EpLibraryStaticPlugin` accepts the entry points directly and reuses -`EpLibraryPlugin`'s factory lifecycle logic without the dynamic-library load and unload steps. - -Rationale: every host — `onnxruntime_test_all`, the Python bindings, `onnxruntime.dll`, and Wasm — gets the provider -with no per-host call site. - -Alternatives considered: - -- CMake-generated registry using an X-macro. Rejected: added build-system machinery for generality that may never be - needed while providers remain in-tree. -- A public `RegisterStaticExecutionProviderLibrary` C API called by each host. Rejected for now: it requires a call - site in every statically linked host and introduces a window in which `GetEpDevices` reports no devices. Revisit - when an out-of-tree provider needs static linking. - -### D2: Entry point names are macro-parameterized - -`onnxruntime/core/providers/webgpu/ep/api.cc` is parameterized by `ORT_PLUGIN_EP_ENTRY_POINT_PREFIX`. - -The shared build must continue to export the unprefixed `CreateEpFactories` and `ReleaseEpFactory`, because -`EpLibraryPlugin::Load` resolves those exact names. The static build emits prefixed variants, for example -`WebGpu_CreateEpFactories`, so that multiple static plugins can coexist in one binary. - -### D3: No new EP library ABI hook for teardown - -All teardown remains in `ReleaseEpFactory`. Process-global shutdown that belongs to the host — currently -`google::protobuf::ShutdownProtobufLibrary()` — is compiled out under `ORT_PLUGIN_EP_OWNS_PROCESS_GLOBALS`. - -Rationale: a new optional entry point cannot be relied upon, because an older ORT would load a newer plugin and -silently ignore it. Behavior that varies with the host version is worse than a compile-time decision. - -The legacy static WebGPU cleanup block in `onnxruntime/core/session/ort_env.cc` is already guarded by -`defined(USE_WEBGPU) && !defined(ORT_USE_EP_API_ADAPTERS)`. The static plugin configuration defines -`ORT_USE_EP_API_ADAPTERS`, so the block compiles out on its own and needs no edit. It must be kept for the default -internal-EP WebGPU build. - -Consequence to test for: static linking has no library unload, so provider process-global state survives -unregistration. Registering, running, unregistering, re-registering, and running again is the primary regression case. - -### D4: One new durable build option - -- `onnxruntime_USE_EP_API_ADAPTERS` is unchanged. It is transitional and is retired when WebGPU moves fully onto the - public plugin API. -- `onnxruntime_WEBGPU_STATIC_PLUGIN` is new and durable. It selects linkage into the host rather than a loadable - module. -- `onnxruntime_WEBGPU_LINKED_INTO_HOST` is derived and used at the linkage sites. - -`onnxruntime_USE_EP_API_ADAPTERS` currently conflates three meanings: compiling against the adapters, building a -separate module, and registering by path in tests. The static plugin configuration answers the first yes and the -other two no, so only the linkage and test sites move to the derived option. The global `add_compile_definitions` in -`cmake/CMakeLists.txt` stays as-is, and no provider sources change. - -`cmake/onnxruntime_providers_webgpu.cmake` gains a third arm. The Emscripten and `onnxruntime_BUILD_CACHE` -`FATAL_ERROR`s narrow to the shared-module case, which is what unblocks ORT Web. - -`build.py` gains `--use_webgpu static_plugin`. - -`onnxruntime_WEBGPU_STATIC_PLUGIN` validates its prerequisites at configure time and fails with `FATAL_ERROR` if -they are not met: it requires `onnxruntime_USE_WEBGPU` and `onnxruntime_USE_EP_API_ADAPTERS`, and it rejects a -minimal build. The last one is a temporary guard rather than a permanent restriction. A minimal build excludes -`core/session/plugin_ep` from `onnxruntime_session_srcs`, and `Environment::CreateAndRegisterStaticPluginEps` is -compiled out with it, but `cmake/onnxruntime_providers_webgpu.cmake` still builds and links the provider. The -combination would therefore produce a binary containing the WebGPU EP that never registers it, with nothing -reporting the problem at configure, build or run time. Allowing minimal builds to use a static plugin EP is -worthwhile and needs the registration path to be available there first; the exclusion is currently justified by -provider-bridge dependencies, and only two of the files under `plugin_ep` are provider-bridge, so the subset may be -separable. Until then a loud configure error is preferable to a silently EP-less binary. - -The CUDA plugin is unaffected: it is gated by `onnxruntime_BUILD_CUDA_EP_AS_PLUGIN` and sets -`ORT_USE_EP_API_ADAPTERS` as a private target compile definition, never referencing the CMake option. CUDA's split -between role (`BUILD_CUDA_EP_AS_PLUGIN`) and mechanism (`ORT_USE_EP_API_ADAPTERS`) is the in-tree template for -WebGPU's eventual cleanup. - -### D5: Test infrastructure treats the library path as optional - -In `onnxruntime/test/unittest_util/test_dynamic_plugin_ep.cc`, `InitializationConfig::ep_library_path` becomes -optional. When absent, library registration is skipped because ORT core has already performed it, and the RAII -registration handle is left empty. The existing handle deleter already tolerates an empty handle. Device selection, -de-duplication, and factory creation are unchanged. - -Virtual devices are enabled through the `allow_virtual_devices` environment configuration entry -(`kOrtEnvAllowVirtualDevices`) supplied at environment creation, rather than the `.virtual` registration-name suffix. -The suffix is unavailable because ORT core chooses the registration name for statically linked providers. - -The `dynamic_plugin_ep_infra` naming becomes inaccurate. Renaming is deferred to a separate mechanical change. - -### D6: Static plugin EPs are registered after the environment is published - -This is the subtlest decision and the one that constrains the others. - -**Problem.** The natural insertion point is next to `CreateAndRegisterInternalEps` inside `Environment::Initialize`. -Registration enumerates devices, which calls provider code: - -``` -OrtEnv::GetOrCreateInstance() - lock(m_) non-recursive - Environment::Create() -> Initialize() - Environment::CreateAndRegisterStaticPluginEps() - Environment::RegisterExecutionProviderLibrary() - EpInfo::Create -> factory GetSupportedDevices() - webgpu::ep::Factory::GetSupportedDevices() provider code - Api().ep.GetEnvConfigEntries() - OrtEnv::TryGetInstance() - lock(m_) self-deadlock -``` - -`Environment::CreateAndRegisterInternalEps` already documents this hazard, and the internal WebGPU factory avoids it -by capturing `allow_virtual_devices` at construction. A plugin factory cannot use that workaround, because the public -API is all it has. - -The problem is not confined to one function. Plugin EPs receive the full `OrtApi`, so any API that resolves the -environment is affected, including `CreateEnv`, which re-enters `GetOrCreateInstance` and deadlocks on the same mutex. - -**Decision.** `OrtEnv::GetOrCreateInstance` publishes `p_instance_` and takes its own reference *before* invoking -`Environment::CreateAndRegisterStaticPluginEps`, and `OrtEnv::m_` becomes a `std::recursive_mutex`. Registration -therefore runs against a fully constructed and published environment. - -Properties: - -- The entire `OrtApi` behaves normally for provider code. `TryGetInstance` finds a published instance, and a - re-entrant `CreateEnv` acquires the recursive lock and increments the reference count. -- No new contract for provider authors, and nothing for a future API author to remember. -- No race. Other threads block on `m_` for the duration, which is already true across `Environment::Create`. -- The reference count cannot reach zero mid-construction, because the creating thread's reference is taken first. -- Provider sources need no conditional compilation for the two linkage modes. - -Costs and limits: - -- Failure during registration requires explicit teardown of the just-published instance. -- A recursive mutex is normally undesirable. Here it states the actual invariant: the environment creation path can - legitimately re-enter the environment accessor on the same thread. -- Callers of `onnxruntime::Environment::Create` do not get static plugin EPs. In the tree this is one production call - site plus tests and orttraining sample binaries. -- Provider code that starts a thread which calls `CreateEnv` and then joins it will deadlock. This is already true of - any code running inside `Environment::Create`. - -The call to `Environment::CreateAndRegisterStaticPluginEps` therefore lives in `ort_env.cc`. Static plugin -registration is a process-singleton concern tied to the `OrtEnv` lifetime, so the singleton wrapper is its correct -home. - -Alternatives considered: - -- Thread-local pointer to the environment under construction, consulted by `GetEnvConfigEntries`. Rejected: it - addresses one function, while the reachable surface is the whole `OrtApi`. -- Defer device enumeration until after environment creation. Rejected: lazy initialization moves registration - failures from environment creation to the first `GetEpDevices` call, which is a worse place to report them. -- Register outside the lock after publishing. Rejected: another thread can observe a published environment whose - static EPs are not yet registered, and avoiding that requires a second lock and a completion barrier. - -`onnxruntime::Environment` remains directly constructible, and tests rely on that. `OrtEnv::GetOrCreateInstance` -returns any existing instance and ignores the logging manager, threading options, and configuration entries passed by -later callers, so a test needing specific threading or logging configuration must bypass it. Those tests are -deliberately opting out of process-singleton semantics and should not receive static plugin EPs. - -### D7: The statically linked build does not use manual C++ API initialization - -`include/onnxruntime/ep/api.h` force-enables `ORT_API_MANUAL_INIT` around its include of `onnxruntime_cxx_api.h`, and -`onnxruntime::ep::ApiInit` calls `Ort::InitApi(ort_api)`. That is required for a plugin EP shared library, which must -not call `OrtGetApiBase()` itself. - -`onnxruntime_cxx_api.h` emits `#pragma detect_mismatch("ORT_API_MANUAL_INIT", ...)` on MSVC, so every translation -unit linked into one binary must agree. The statically linked plugin EP is linked with ORT core and with test code, -neither of which uses manual initialization, so forcing it on produces `LNK2038` for every EP object file. - -The build therefore defines `ORT_PLUGIN_EP_STATICALLY_LINKED` on the statically linked plugin EP target. Under that -macro `ep/api.h` includes `onnxruntime_cxx_api.h` unmodified and `ApiInit` skips `Ort::InitApi`. The C++ API then -default-initializes from `OrtGetApiBase()->GetApi(ORT_API_VERSION)`, which resolves in-process and yields the same -`OrtApi` that `ApiInit` would have installed, because the EP and ORT are the same binary and therefore the same -version. `onnxruntime::ep::ApiPtrs` is still populated from the `OrtApiBase*` that ORT passes to `CreateEpFactories`, -so the EP's own API access is unchanged. - -The macro is generic rather than WebGPU-specific because `ep/api.h` is shared with the CUDA plugin EP. - -## Implementation order - -1. `EpLibraryStaticPlugin`, factoring the shared factory lifecycle out of `EpLibraryPlugin`. -2. The static registry and `Environment::CreateAndRegisterStaticPluginEps`, with the `ort_env.cc` - publish-then-register change. -3. `ep/api.cc` entry point prefixing and host-owned process-global guards. -4. Build option and `build.py` changes. -5. Test infrastructure changes. -6. `ORT_PLUGIN_EP_STATICALLY_LINKED` in `include/onnxruntime/ep/api.h`, per D7. -7. Applying the WebGPU plugin EP test definitions to `onnxruntime_provider_test` as well as `onnxruntime_test_all`, - via a shared `onnxruntime_set_webgpu_plugin_ep_test_definitions` cmake function. `onnxruntime_provider_test` holds - the operator tests, so without this the plugin path has no operator test coverage. It also needs - `ORT_UNIT_TEST_HAS_WEBGPU_STATIC_PLUGIN_EP` in its own right, because that macro is what enables the - `allow_virtual_devices` environment configuration entry at environment creation in `test_main.cc`. - -## Follow-up cleanup - -`Environment::CreateAndRegisterInternalEps` stays where it is, at the end of `Environment::Initialize`. Internal EP -factories are ORT-core code that does not re-enter the `OrtEnv` singleton, so they are not subject to the constraint -in D6. Moving them would also remove EP devices from every directly constructed `Environment`, and `InferenceSession` -reads `Environment::GetOrtEpDevices` directly. For static plugin EPs the resulting gap is unimplemented new -functionality; for internal EPs it would be a silent regression. - -The `allow_virtual_devices` parameter threaded through `EpLibraryInternal::CreateInternalEps` exists only for the -internal WebGPU EP factory, which cannot query the environment at `GetSupportedDevices` time. The comment in -`CreateAndRegisterInternalEps` documents that constraint. Both can be deleted once WebGPU is no longer an internal EP: -the remaining internal EPs are CPU, kept deliberately as a special case, and DML, which is no longer maintained. -Neither takes the parameter. In a build that defines `ORT_USE_EP_API_ADAPTERS` this is already true, since the -internal WebGPU EP is compiled out and the parameter reaches `ORT_UNUSED_PARAMETER`. - -This does not affect the `.virtual` registration-name suffix or the `allow_virtual_devices` environment -configuration entry, which remain in use for dynamically registered plugin libraries. - -## Risks and open questions - -- Whether adapter-compiled WebGPU objects link into ORT core without duplicate-symbol or ODR problems. One real - instance was found and resolved: the `ORT_API_MANUAL_INIT` `detect_mismatch` guard, see D7. Otherwise resolved: - `onnxruntime_test_all` and `onnxruntime_provider_test` both link cleanly against the statically linked EP. -- Provider process-global state surviving unregistration, since static linking has no unload event. Covered by the - re-registration regression case in D3. Not yet exercised: `OrtEnv` is a reference-counted process singleton and the - unit test main holds a reference for the lifetime of the process, so a second `Environment` construction (and - therefore a second `CreateEpFactories` call) is not reachable from the existing test binaries. This needs either a - dedicated single-test process or a test that fully releases and recreates the environment. -- Whether any shipped ORT Web configuration uses an extended-minimal build, which excludes plugin EP infrastructure - entirely. -- Whether any `std::condition_variable` is paired with `OrtEnv::m_`, which would require `condition_variable_any` - under a recursive mutex. -- The existing rejection of `onnxruntime_BUILD_DAWN_SHARED_LIBRARY` together with the adapters. Initially left - rejecting both plugin kinds. -- Binary size and dead-code elimination for statically linked providers, per the workstream completion criteria. - -## Validation status - -Verified on Windows with `--use_webgpu static_plugin`, Debug: - -- `onnxruntime_test_all` and `onnxruntime_provider_test` build and link. -- The statically linked WebGPU plugin EP is registered by ORT core during `Ort::Env` creation, and its `OrtEpDevice` - is enumerated and selectable, confirming the D6 publish-then-register ordering resolves the self-deadlock. -- `InferenceSessionTests.WebGpuVirtualDeviceCompileOnlyEndToEnd` and - `InferenceSessionTests.WebGpuVirtualDeviceRejectedWithoutCompileOnly` pass, which covers session creation and EP - instantiation end to end through the public V2 API on the static plugin path. - -Operator execution was initially not verifiable locally: the original development machine was a virtual machine with -no real GPU, so only the virtual WebGPU device was surfaced and non-compile-only sessions were rejected up front. This -was confirmed to be a property of the host rather than of static linking by building the shared library plugin -configuration (`--use_webgpu shared_lib`) on the same host and running the same tests: it passed and failed exactly -the same set (`WebGpuVirtualDeviceCompileOnlyEndToEnd` and `WebGpuVirtualDeviceRejectedWithoutCompileOnly` pass; -`WebGpuCompileOnlySkipsFinalization` and `TestStrictShapeInference` fail with the identical virtual-device rejection -message). - -This gap has since been closed on a machine with a real GPU (NVIDIA RTX 5060 Ti). The `static_plugin` and internal-EP -configurations were built from the same tree, RelWithDebInfo, differing only in the EP path, and their test results -compared: - -- `onnxruntime_provider_test`: 0 failures in both. **Across the 5894 tests common to both builds there were zero - status differences.** -- `onnxruntime_test_all`: 1965 passed in both, no differences. All three `InferenceSessionTests.WebGpu*` pass on the - static plugin build, including `WebGpuCompileOnlySkipsFinalization`, confirming its earlier failure was indeed a - property of the GPU-less host. - -### Known test coverage difference on the plugin path - -`cmake/onnxruntime_unittests.cmake` gates the `test/providers/webgpu/*` sources on -`onnxruntime_USE_WEBGPU AND NOT onnxruntime_USE_EP_API_ADAPTERS`, so those tests are **absent** from any adapter -build, including `static_plugin`. They are white-box tests that include internal WebGPU EP headers and therefore -cannot compile against the adapter boundary by construction. - -Of the 44 tests this excludes, 16 are `DISABLED_` and never run, and most of the rest assert on EP internals -(`WebGpuContextTest`, `ActivationCacheKeyTest`). However, 12 of them — `HardSwish_WebGPU` and `MatMul2BitsWebGpu` — -are genuine operator tests, and they currently run **only** on the internal EP path. This is a known coverage gap of -the plugin path. Closing it would mean rewriting them as EP-agnostic operator tests so both paths execute them; that -is deliberately left out of this change to keep it focused. - -Separately, the `InferenceSessionTests.WebGpu*` virtual-device tests are deliberately *paired* rather than shared -between the two builds, since each drives a different factory. `WebGpuVirtualDeviceCompileOnlyEndToEnd` and -`WebGpuVirtualDeviceRejectedWithoutCompileOnly` are the plugin-build counterparts of `WebGpuEpFactoryVirtualDevice` -and `WebGpuEpFactoryRejectsVirtualDeviceWithoutCompileOnly`. Only `WebGpuCompileOnlyUsesNoOpAllocator` has no -counterpart, because it asserts an internal allocator *type* through `dynamic_cast`, which is inherently -unobservable across the ABI boundary. - -The shared library plugin configuration was also re-verified after the D7 change, since `include/onnxruntime/ep/api.h` -is shared with the CUDA plugin EP. It builds, and `onnxruntime_providers_webgpu.dll` still exports exactly the -unprefixed `CreateEpFactories` and `ReleaseEpFactory` required by `EpLibraryPlugin::Load`, per D2. - -### CI coverage - -No pre-existing CI leg built this configuration, since both new CMake options default to `OFF`. Two jobs were added: - -- `webgpu_static_plugin_build_x64_RelWithDebInfo` in `.github/workflows/windows_webgpu.yml`, which builds and runs the - tests on a GPU-equipped pool. This closes the operator execution gap above. Note that it must not set - `onnxruntime_BUILD_DAWN_SHARED_LIBRARY=ON`, which is incompatible with `onnxruntime_USE_EP_API_ADAPTERS`. -- `build-linux-webgpu-static-plugin-x64-release` in `.github/workflows/linux_webgpu.yml`, build only, mirroring its - sibling job. This is the only GCC coverage of the EP adapters, and the only coverage of the `onnxruntime_python.cmake` - change, hence `--build_wheel` is kept. - -The first run of the Linux job surfaced `-Werror=maybe-uninitialized` errors. The EP adapter `OpKernelInfo` defines its -attribute accessors inline in the header, so GCC can see their failure and exception paths, whereas the in-tree -`OpKernelInfo` hides them behind an out-of-line definition. Any value that is only read when the attribute lookup -succeeded must be value-initialized. - -More generally, this job is the first time the EP adapters have ever been compiled with GCC, and it has surfaced -several successive batches of `-Werror` diagnostics — including false positives inside `absl::InlinedVector` and, in -optimized builds, `-Werror=array-bounds` on `std::string` concatenation chains that the adapter headers cause to be -inlined more aggressively. Because ninja stops at the first failure and reports only the targets already in flight, -**each CI run reveals just one batch**. When touching this configuration, enumerate the full set locally in a single -pass — an AlmaLinux 8 container with `gcc-toolset-14` reproduces the CI compiler exactly — rather than discovering -them one CI round trip at a time. Fixes that are correct on their own merits are applied at the source; irreducible -false positives use a suppression scoped to the `onnxruntime_providers_webgpu` target, GCC only, and conditional on -`onnxruntime_USE_EP_API_ADAPTERS`, so non-adapter builds stay strict. This mirrors what the CUDA plugin EP does in -`cmake/onnxruntime_providers_cuda_plugin.cmake`. - -That full local enumeration has since been done, and it is worth recording what it found: of the ~340 warnings GCC 14 -emits across the whole `static_plugin` build, **exactly one is attributable to ORT's own source** — the -`-Warray-bounds` false positive in `contrib_ops/webgpu/quantization/matmul_nbits_mlp.cc`, fixed by building the -expression with `MakeString` instead of a `std::string` concat chain. Everything else lives in Dawn, ONNX, -flatbuffers, or the telemetry dependency, in targets that do not use `-Werror`. The practical lesson is that the -adapters do not make ORT's code broadly warning-prone; they perturb inlining at a small number of specific sites, -so a targeted source fix is almost always available and preferable to a blanket target-wide suppression. - -When validating such a fix locally, always rebuild the *unfixed* source through the same harness first and confirm it -still errors. A harness that silently fails to exercise the warning is indistinguishable from a successful fix. - -### Emscripten and ORT Web - -Making the statically linked WebGPU EP reachable through the plugin path is what motivates this design, so the -Emscripten configuration was built before landing the change: `--build_wasm --enable_wasm_simd ---enable_wasm_threads --use_webgpu static_plugin --use_webnn --target onnxruntime_webassembly`. It configures, -compiles and links cleanly, producing `ort-wasm-simd-threaded.asyncify.{mjs,wasm}`. - -`static_plugin` is in fact the only plugin kind Emscripten supports — `cmake/onnxruntime_providers_webgpu.cmake` -raises a `FATAL_ERROR` for `shared_lib` there, since there is no runtime library to load. - -No CI leg covers this yet. The WASM WebGPU jobs pass a bare `--use_webgpu`, which defaults to `static_lib` -(`tools/ci_build/build_args.py` uses `const="static_lib"`), so they still build the internal EP. This change makes -the configuration available to ORT Web; switching ORT Web over to it is separate work, and has prerequisites that -this change does not address. See [ORT Web migration](#ort-web-migration). - -A green build is *not* sufficient evidence that the registration is live. `CreateStaticPluginEpLibraries()` in -`onnxruntime/core/session/plugin_ep/ep_static_plugins.cc` is guarded by -`#if defined(USE_WEBGPU) && defined(ORT_WEBGPU_STATIC_PLUGIN)`; if that define does not reach ORT core the function -still compiles, simply returning an empty vector, and the build is just as green while registering nothing. The -check that actually discriminates is at the symbol level — ORT core must hold an *undefined* reference to the -prefixed entry points, resolved by the provider library: - -``` -llvm-nm libonnxruntime_session.a | grep WebGpu_ # U WebGpu_CreateEpFactories / U WebGpu_ReleaseEpFactory -llvm-nm libonnxruntime_providers_webgpu.a | grep WebGpu_ # T WebGpu_CreateEpFactories / T WebGpu_ReleaseEpFactory -``` - -Both were observed on the Emscripten build, which confirms the guard was live, the D3 entry-point prefixing worked, -and the link resolved core's reference against the provider. That check validates build, link and registration -wiring; runtime behaviour was established separately by the ORT Web prototype described under -[ORT Web migration](#ort-web-migration). - -A practical build note: the WASM build needs `node` on `PATH` — `node_helper.cmake` does a hard `find_program` and -fails configure without it. emsdk bundles a suitable one under `cmake/external/emsdk/node//bin`. - -## ORT Web migration - -The intended end state is that the WebGPU EP is always a plugin EP and `--use_webgpu static_lib` goes away -entirely, ORT Web included. Building the EP into the WASM binary as a static plugin, which this change enables, is -only the first half of that. The second half is *selection*, and two gaps stand between the current state and a -working ORT Web on the plugin path. Neither is visible to the linker, so the symbol-level evidence above does not -speak to either. - -Both gaps have since been closed by a throwaway prototype, which ran the full `js/web` WebGPU operator suite in -Edge against a WASM `static_plugin` binary: 2152 tests, all passing. The design below is therefore validated end -to end rather than merely plausible; what remains is productionization, not discovery. The prototype's findings are -recorded inline in the steps that follow. - -**ORT Web selects the EP by name, and that name resolves only to the internal EP.** -`js/web/lib/wasm/session-options.ts` maps the `'webgpu'` backend to `epName = 'WebGPU'`, which reaches -`OrtAppendExecutionProvider` in `onnxruntime/wasm/api.cc` and from there -`OrtApis::SessionOptionsAppendExecutionProvider`. That API is a closed table of hardcoded EP names in -`onnxruntime/core/session/provider_registration.cc`, and its `EpID::WebGPU` case is compiled out under -`ORT_USE_EP_API_ADAPTERS`. On a `static_plugin` build the EP therefore registers at `OrtEnv` creation and is then -unselectable from JS: the call returns "WebGPU execution provider is not supported in this build". This is by -design rather than an oversight — the name-based API is the internal-EP mechanism, and plugin EPs are selected -through `RegisterExecutionProviderLibrary` plus `OrtEpDevice`-based selection — but it does mean ORT Web needs new -plumbing rather than a recompile. - -**No WebGPU `OrtEpDevice` exists under Emscripten.** Emscripten was neither `WIN32`, `LINUX` nor `APPLE`, so -`cmake/onnxruntime_common.cmake` fell through to `core/platform/device_discovery_default.cc`, which discovers only -a CPU device. `Factory::GetSupportedDevicesImpl` in `onnxruntime/core/providers/webgpu/ep/factory.cc` creates an -`OrtEpDevice` only for devices of type `OrtHardwareDeviceType_GPU`. The one fallback, the virtual GPU device in the -same function, is opt-in through the `allow_virtual_devices` environment configuration entry and is deliberately -registered without allocator info, because it exists to back a device-free compile-only session. So even after -selection plumbing is added, `GetEpDevices` would offer nothing for JS to select. - -A workable order: - -1. Make WebGPU visible as a device under Emscripten by adding an Emscripten device discovery implementation. - `cmake/onnxruntime_common.cmake` gains an `Emscripten` arm selecting a new - `core/platform/emscripten/device_discovery.cc`, which reports the CPU device via `GetCpuDeviceFromCPUIDInfo()` - plus a GPU `OrtHardwareDevice` when a synchronous `EM_ASM_INT` check finds `navigator.gpu`. No change to - `Factory::GetSupportedDevicesImpl` is needed: its existing loop filters on `OrtHardwareDeviceType_GPU`, so a - discovered GPU flows through and picks up both allocator infos unmodified. - - The prototype first tried the other option — having the factory synthesize the device itself through - `CreateHardwareDevice` — and it worked, but it is the wrong layer. `GetSupportedDevices` documents its `devices` - argument as "the `OrtHardwareDevice` instances that are available", so that array is the source of truth, and - `CreateHardwareDevice` is documented for devices with no hardware behind them ("e.g., virtual"). A browser GPU - is real hardware that ORT's discovery layer merely failed to enumerate, so the producer is the place to fix it. - Leaving it in the factory would also oblige every future Emscripten plugin EP to repeat the same workaround. - - A capability check is a legitimate form of discovery here. `core/platform/apple/device_discovery.cc` sets the - precedent: it enumerates nothing, it asserts that one GPU exists on Apple Silicon and hardcodes the vendor id. - `navigator.gpu` is a stronger signal than that — an actual runtime capability probe, and the same one ORT Web's - `initEp` uses to decide whether WebGPU is usable. `EM_ASM` from ORT core is likewise established practice - (`core/graph/model.cc`, `core/framework/external_data_loader.cc`), not just an EP-layer habit. - - The honest limitation is that no adapter information is available. `_OrtInit` creates the `OrtEnv` and therefore - triggers discovery, and it runs before `navigator.gpu.requestAdapter()` in ORT Web's `initEp`, so `vendor_id` - and `device_id` are left at 0. That is acceptable: Apple reports no device id either, and browsers mask - `GPUAdapterInfo.vendor` for fingerprinting resistance regardless. The device is deliberately *not* tagged with - `kOrtHardwareDevice_MetadataKey_IsVirtual`, which is what keeps it distinct from the virtual device — that path - is allocator-less by design and cannot back a real session. - - A later change on `main` added a second fallback to `GetSupportedDevicesImpl` alongside the virtual device: when - no GPU device is discovered, `allow_software_adapter` (opt-in via the `kAllowSoftwareAdapterEnvironmentVariable` - environment variable) advertises WebGPU against the *CPU* `OrtHardwareDevice`. It does not change the analysis - above. Both upstream fallbacks work around a device the discovery layer failed to enumerate; they are opt-in - precisely because misreporting a device is not safe to do by default. Under Emscripten the device is genuinely - there and cheaply detectable, so fixing discovery keeps the default path correct and needs no opt-in. The - primary matching loop that our discovered GPU device flows through is unchanged by that work. - - One accepted side effect: `ep_library_internal.cc` creates an internal `WebGpuEpFactory` in a - `USE_WEBGPU && !ORT_USE_EP_API_ADAPTERS` build, so today's `static_lib` ORT Web build starts producing an - `OrtEpDevice` where it previously produced none. That build selects by name and is slated for removal, so the - change is benign. - -2. Expose plugin EP selection in the WASM C API. `g_env` is already available in `api.cc`, so `GetEpDevices` and - `SessionOptionsAppendExecutionProvider_V2` are both reachable. A narrow helper that appends by EP name and - resolves the device internally avoids marshalling arrays of device pointers across the JS boundary. The new - entry point needs to be exported from `cmake/onnxruntime_webassembly.cmake`. - - *Prototype result:* the name-resolving helper is the right shape. Note that exporting it takes three edits, not - one: `EMSCRIPTEN_KEEPALIVE` in `api.h`, the `JSPI_EXPORTS` list in `cmake/onnxruntime_webassembly.cmake` for - JSPI builds, and the `wrapAsyncAPIs` list in `onnxruntime/wasm/pre-async.js` for Asyncify builds. Omitting the - last one is the easy mistake, and Asyncify is the default variant. Reporting matters too: on no match the helper - should raise a status naming the EPs that *are* registered, which is what makes a misconfiguration diagnosable - from JS through `checkLastError`. - -3. Switch `js/web` to the new call. The `BUILD_DEFS.DISABLE_WEBGPU` branch that falls back to the JS EP is - unaffected. Confirm that the EP options ORT Web passes today — `deviceId`, `webgpuInstance`, `webgpuDevice`, - `preferredLayout` and the buffer cache modes — still reach the provider, since `Factory::CreateEpImpl` obtains - them from the session config options. - - *Prototype result:* the options do reach the provider unchanged, and the reason is worth recording. Both paths - derive their config prefix from `OrtSessionOptions::GetProviderOptionPrefix` keyed on the EP name, so V2 lands - the options under exactly the same `ep.webgpuexecutionprovider.*` keys the name-based path used. One JS change - is required though: `session-options.ts` passes the short name `'WebGPU'`, whereas `OrtEpDevice` selection - matches the canonical `'WebGpuExecutionProvider'` returned by `EpDevice_EpName`. - - The productionization question this raises is *conditionality*: is it safe to switch to V2 unconditionally - whenever WebGPU is enabled? It is, provided the WASM build flips at the same time. `BUILD_DEFS.DISABLE_WEBGPU` - is defined in `js/web/script/build.ts` as exactly `!USE_WEBGPU_EP`, so the bundle is already specialized per - WASM flavour by `--webgpu-ep` — the JSEP bundle takes the other branch entirely and is unaffected. Flipping the - WebGPU WASM builds to `static_plugin` (step 4) therefore makes `!DISABLE_WEBGPU` imply `static_plugin` by - construction, and no additional build define or runtime capability probe is needed. What is *not* safe is - landing the JS change without the build change, since the two must move together. - -4. Flip the ORT Web WASM builds to `static_plugin` and add a CI leg. The two builds are - `Build (simd + threads + WebGPU experimental)` and its JSPI sibling in - `.github/workflows/linux-wasm-ci-build-and-test-workflow.yml`, both of which pass a bare `--use_webgpu` and so - get `static_lib` from `build_args.py`'s `const="static_lib"`. Neither uses `--minimal_build`, so the - minimal-build `FATAL_ERROR` guard does not stand in the way of this migration. No pipeline builds WASM with - `static_plugin` today, so without a leg this path stays validated at the symbol level only. - - *Prototype note:* the `js/web` test runner cannot currently exercise this configuration end to end, because - `script/test-runner-cli.ts` spawns `script/build` with only `--bundle-mode` and does not forward `--webgpu-ep`. - Setting `npm_config_webgpu_ep` in the environment works as a stopgap, since `script/build.ts` reads it, but a CI - leg should forward the flag properly. Note also that without `--webgpu-ep` the `webgpu` backend silently routes - through JSEP instead, so a leg that omits it would pass while testing nothing relevant. - -### Interaction with the ORT Web reduced-size build options - -The ORT Web WASM builds are size-sensitive and pass a set of size-reduction flags that the local prototype builds -did not. `linux-wasm-ci-build-and-test-workflow.yml` applies `--disable_ml_ops --disable_generation_ops ---disable_types string float4 float8 optional sparsetensor --include_ops_by_config -onnxruntime/wasm/reduced_types.config --enable_reduced_operator_type_support`, and `web.yml` adds `--disable_rtti` -for the release job. Since the migration changes how the WebGPU EP is compiled and linked, each of these was -checked against the plugin path. - -**None of them need adapting.** The findings: - -- *Compile definitions reach the WebGPU target in every mode.* `onnxruntime_providers_webgpu` is created through - `onnxruntime_add_static_library` / `onnxruntime_add_shared_library_module`, both of which call - `onnxruntime_configure_target` → `onnxruntime_set_compile_flags`. That function is where every `DISABLE_*` and - `REDUCED_OPS_BUILD` definition is applied, so the `static_lib`, `static_plugin` and shared-library builds all - receive an identical set. There is no propagation gap introduced by the plugin boundary. - -- *Operator and type reduction never applied to WebGPU, before or after.* - `op_registration_utils.get_kernel_registration_files()` hardcodes the CPU registration files (plus CUDA when - requested); WebGPU is not in the list, so `--include_ops_by_config` does not rewrite WebGPU kernel - registrations. `onnxruntime/wasm/reduced_types.config` is `!no_ops_specified_means_all_ops_are_required` — it - performs global *type* reduction only, with no op exclusion — and that mechanism works through the - `op_kernel_type_control` macros, which neither `core/providers/webgpu` nor `contrib_ops/webgpu` uses. Its saving - is entirely CPU-EP-side and is unchanged by the plugin switch. This is a pre-existing gap, not a regression. - -- *The type and ML/generation op flags are inert for WebGPU.* No source under `core/providers/webgpu` or - `contrib_ops/webgpu` references `DISABLE_ML_OPS`, `DISABLE_GENERATION_OPS`, `DISABLE_SPARSE_TENSORS`, - `DISABLE_OPTIONAL_TYPE`, `DISABLE_FLOAT8_TYPES`, `DISABLE_FLOAT4_TYPES` or `DISABLE_STRING_TYPE`. - -- *`--disable_rtti` is safe on the plugin path.* Neither `include/onnxruntime/ep` (the EP API adapters) nor - `onnxruntime/core/session/plugin_ep` uses `dynamic_cast` or `typeid`. - -One consequence is worth recording for the *shared library* plugin EP, which is out of scope here but shares this -code. `onnxruntime_c_api.h` contains no `DISABLE_*` guards, so the `OrtApi` struct layout is invariant under these -flags and the ABI is stable. `onnxruntime_cxx_api.h` does guard some `Ort::Value` members (the sparse tensor -methods) — an EP DLL built with a different `--disable_types` than its host would therefore see a different C++ -header surface, but since those wrappers are inline over stable function pointers this is a source-compatibility -concern rather than an ABI one. For `static_plugin` everything is compiled in one tree with one set of defines, so -the question does not arise. - -**What is *not* covered by any existing flag** is the plugin machinery itself. Relative to `static_lib`, the -`static_plugin` build adds `core/providers/webgpu/ep/*` (excluded from the source list in the `static_lib` branch -of `onnxruntime_providers_webgpu.cmake`), inlines the header-only EP API adapters into every WebGPU translation -unit, and reaches the `OrtGraph` / `OrtNode` C API graph views through `OrtEp::GetCapability`. Note that -`core/session/plugin_ep` is *not* part of that delta: `onnxruntime_session.cmake` only excludes it for -`onnxruntime_MINIMAL_BUILD`, so today's WASM builds already compile it in. - -That also means the only existing lever that trims plugin-EP machinery is `--minimal_build`, which -`static_plugin` currently rejects outright. If measurement shows the size delta matters, lifting that restriction -(already listed as a follow-up) is the mechanism to reach parity — not a new size-reduction flag. - -#### Measured size comparison - -Both modes were then built locally with the exact CI flag set (`Release`, SIMD + threads, asyncify, WebNN on, -`--disable_rtti`, `--enable_wasm_api_exception_catching`, and the full reduced-size set above) and the artifacts -compared: - -| Artifact | `static_lib` | `static_plugin` | Delta | -| --- | ---: | ---: | ---: | -| `ort-wasm-simd-threaded.asyncify.wasm` | 26,550,879 B | 26,488,911 B | **−61,968 B (−0.23%)** | -| `ort-wasm-simd-threaded.asyncify.mjs` | 53,249 B | 53,249 B | 0 | - -(Both binaries rebuilt after merging `main`. An earlier pre-merge pair measured −52,978 B / −0.20%; the merge added -kernels to both builds and did not change the relationship.) - -**There is no size regression** — `static_plugin` is marginally smaller. Both configurations compiled cleanly with -every reduced-size flag, which empirically confirms the analysis above. The plugin build plausibly comes out ahead -because it drops the internal `IExecutionProvider` / `KernelRegistry` glue and the built-in `WebGpuEpFactory` in -`ep_library_internal.cc` in favour of the header-only adapters, but that attribution is not separately measured. -For reference, the same `static_plugin` configuration without any reduced-size flags is 40.9 MB, so those flags are -worth ~15 MB and remain essential regardless of EP mode. - -#### `--enable_wasm_api_exception_catching` and the EP API adapters - -The one option that *did* interact with the plugin path is `--enable_wasm_api_exception_catching`. It is not a -size-reduction flag in the same sense as the others, but it is part of the same CI flag set and it exposed a real -bug. - -`onnxruntime_webassembly.cmake` compiles only `wasm/api.cc` and `core/session/onnxruntime_c_api.cc` with -`-sDISABLE_EXCEPTION_CATCHING=0`; every other translation unit keeps Emscripten's default, where `catch` clauses -are compiled so that they never match. Exceptions still propagate — they are simply only catchable at the C API -boundary. - -The EP API adapters in `include/onnxruntime/ep/adapter/` are header-only and are therefore inlined into WebGPU -translation units that have catching disabled. `OpKernelInfo::GetAttr()` / `GetAttrs()` were implemented as a -`try` around the throwing `Ort::` C++ wrappers, converting `Ort::Exception` into a `Status`. That makes a *missing -optional attribute* — ordinary control flow for `GetAttrOrDefault()` — depend on catching an exception. With -catching disabled the `catch` never ran, so the exception escaped `GetAttrOrDefault()` entirely and surfaced from -session creation as e.g. `ERROR_CODE: 6, ERROR_MESSAGE: No attribute with name:'extrapolation_value'is defined.` - -This reproduced as 949 `suite0` failures on the reduced-size `static_plugin` build while the same build passed on -the CPU backend, the reduced-size `static_lib` build passed, and the full-op `static_plugin` build passed — the -failure required plugin + WebGPU + API-only exception catching together. - -The fix is to not use exceptions for control flow in the adapters: `GetAttr()` / `GetAttrs()` now call the -non-throwing `OrtApi::KernelInfoGetAttribute*` functions directly and convert the returned `OrtStatus*` into a -`Status`. **General rule: header-only EP API adapter code must not rely on catching exceptions**, because a plugin -EP may be compiled with exception catching disabled. The remaining `catch` blocks in `include/onnxruntime/ep` -(`common.h`, `get_capability_utils.h`, `adapter/kernel_registry.h`) sit on genuinely exceptional callback -boundaries rather than on control-flow paths; when catching is disabled those exceptions still reach the outer C -API boundary and are reported, only with a less specific message. - -With that fix the reduced-size `static_plugin` build passes `suite0` on the WebGPU backend in full: 2152 tests, -all passing, matching the `static_lib` control run on the identical flag set. - -#### Measured performance comparison - -The same two binaries were compared on inference performance in Edge, using the test runner's `--perf` mode -(`test-runner-cli model -b=webgpu -e=edge -P=`) against two synthetic models chosen to isolate the two -places the plugin path could plausibly cost something: - -- **dispatch-bound** — 300 chained tiny elementwise ops (`Mul`/`Add`/`Sub`) on a `[1, 1024]` tensor. Almost no GPU - work per node, so the run time is dominated by per-node host-side work, which is exactly what now goes through - the C API adapters. -- **compute-bound** — 16 chained `[512, 512]` `MatMul`s. Host-side per-node cost is negligible against the GPU - work. - -Three runs per model per build. **The two builds must be sampled interleaved** — one round is -`plugin/dispatch, plugin/compute, lib/dispatch, lib/compute`, repeated — rather than all samples of one build -followed by all samples of the other. This machine drifts: a build that has just finished, a warm browser, or a -warm GPU shifts timings by more than the effect being measured, and sequential sampling aliases that drift onto -the build under test. The sequential methodology used for the first pass understated a real regression by nearly -an order of magnitude (see below). The table reports the median of the three per-run P50 values. - -| Metric | `static_lib` | `static_plugin` | Delta | -| --- | ---: | ---: | ---: | -| dispatch-bound, per-run P50 | 11.20 ms | 12.00 ms | +0.8 ms (+7%) | -| compute-bound, per-run P50 | 7.70 ms | 7.80 ms | +0.1 ms (within noise) | - -Reading of these numbers: - -- **Execution is not measurably slower once GPU work dominates.** The compute-bound samples overlap - (`static_plugin` 7.80 / 7.80 / 7.40 against `static_lib` 7.90 / 7.00 / 7.70 — the plugin build's best sample - beats two of the three control samples), so the difference is not resolvable at this sample count. -- **A dispatch-bound graph pays roughly 7%.** That is the cost of routing per-node kernel work through the C API - adapters instead of a direct in-process virtual call. Unlike the compute-bound case this one does separate - cleanly — the plugin build's slowest control sample is still faster than its own fastest sample. It only shows - up when nodes do essentially no GPU work, which is the worst case by construction rather than a representative - one. Amortized over ~300 nodes it is on the order of 3 µs per node. -- **Session creation costs about 3%** (measured pre-merge: 1136 → 1172 ms and 1250 → 1290 ms for the two models). - This covers plugin EP registration, `GetCapability` and kernel creation through the C API. The absolute figure - is dominated by fetching and parsing the model, so the true relative cost of the plugin machinery within session - creation is higher than 3% — but it is tens of milliseconds once, not per inference. - -Two cautions on methodology: - -- The *total* wall time the test runner prints for `suite0` is not a usable metric. It varied between 3 min 15 s - and 4 min 04 s for the **same** `static_lib` binary across two runs. Only the second figure it prints — net test - time, excluding the per-model `before all` session-creation hooks — is stable enough to compare, and by that - measure the two builds are equal. An early single-sample comparison of the total figure suggested a 29% - regression that does not exist. -- Timings taken with `--webgpu.profiling.mode=default` are inflated by the timestamp queries and must never be - compared against timings taken without it. - -#### The shared allocator must defer zero-initialize submission during a run - -Interleaved sampling exposed a genuine, code-level regression in the plugin path that sequential sampling had -hidden: the compute-bound model ran at 9.90 ms against the control's 7.30 ms, **+36%**, with no overlap between -the two sample sets. That the dispatch-bound model — 300 nodes against the compute model's 16 — did *not* show a -proportional cost ruled out a fixed per-node or per-run overhead and pointed at something scaling with bytes -allocated rather than with node count. - -The cause was buffer zero-initialization. `BufferManager::Create` clears a buffer recycled from the cache, and -`GpuBufferAllocator` takes a `should_submit_zero_initialize` predicate deciding whether that clear is submitted -before `Create` returns. During `Session::Run` it must **not** be: the clear is encoded into the run's own command -encoder and rides along with the run's submission, preserving dispatch batching. Outside a run there is no -upcoming submission to piggyback on, so the clear must be submitted immediately. `WebGpuExecutionProvider` and the -plugin factory's per-EP allocator both passed `!IsRunActive()`, but the factory's **shared** allocator — the one -backing intermediate tensors — hardcoded `true`, forcing a queue submit for every intermediate buffer mid-run. -The compute model allocates sixteen 1 MB intermediates, so it paid sixteen forced submits per inference. - -The shared allocator is created by `CreateAllocatorImpl` before any EP instance exists, so it had no EP to ask — -which is why the predicate was stubbed out rather than wired up. The fix moves the run-active flag onto -`WebGpuContext`, which is both reachable from the allocator and the more correct owner: it holds -`current_command_encoder_` / `current_compute_pass_encoder_`, the very state that batching depends on. -`WebGpuExecutionProvider` sets it in `OnRunStart` / `OnRunEnd` and its `IsRunActive()` now delegates to the -context, so all three allocator sites read one source of truth. With the fix the compute-bound gap disappears into -noise, as the table above shows. - -This bug predates the ORT Web migration and affects any consumer of the WebGPU plugin EP, not just the web build; -it is a candidate to land separately from the static-plugin work. - -It was present in the pre-merge measurements too, which reported only +4% for this model. The reason is that the -cost is state-dependent rather than constant: `BufferManager::Create` only clears — and therefore only flushes — -a buffer it recycles from its cache, so the penalty depends on cache occupancy and appears as a wide, roughly -bimodal distribution rather than a fixed offset. Sequential sampling of the *same* defective binary produced -per-run P50s spanning 7.70 ms to 10.20 ms, a range that brackets the pre-merge figure. The merge itself did not -introduce or amplify the defect: of the three merged commits touching this path, #32272 only adds an early return -when an allocator is already registered (not reached with a single device), and #32269's subgroup-matrix MatMul is -gated on the F16 8x16x16 config, so the f32 compute-bound model never takes it. The lesson is about sample count -and interleaving on a high-variance metric, not about the merge. - - - -> Should static factories be registered before environment creation or through environment construction options? - -Neither. They are registered by ORT core during `OrtEnv` creation, after the environment is constructed and -published. See D6. diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md deleted file mode 100644 index 04309e6f32feb..0000000000000 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md +++ /dev/null @@ -1,521 +0,0 @@ -# Static plugin EP registration — work log - -Working notes for the branch `edgchen1/webgpu_static_plugin`. This is a scratch handoff document, not a -design document: the design and its rationale live in -[`static_plugin_ep_registration_design.md`](./static_plugin_ep_registration_design.md), which is the -authoritative reference for decisions D1–D7. **Delete this file before opening the PR.** - -The purpose of this log is to let someone pick the work up on a different machine — in particular a machine -with a real GPU, which the original development machine did not have. - -## Goal - -Make the statically linked WebGPU EP build go through the **plugin EP** path rather than the internal-EP path, -via a generic "static plugin EP" registration mechanism in ORT core. The same EP sources -(`onnxruntime/core/providers/webgpu/ep/*`) then serve both a dynamically loaded plugin DLL and a build that is -statically linked into the host. The latter is what unblocks ORT Web / Emscripten, where shared library modules -are not possible. - -Whether the EP is dynamically or statically **linked** is deliberately separate from whether it is a **plugin**. -Both linkage modes are supported. - -## Status - -The implementation is complete and committed. CI coverage was added for the new configuration, which had none. - -| Area | State | -| --- | --- | -| Implementation | Done | -| Windows build + GPU tests (`--use_webgpu static_plugin`) | Green in CI | -| Shared library plugin build (`--use_webgpu shared_lib`) | Green in CI, no regression from D2/D7 | -| CUDA plugin EP (shares `include/onnxruntime/ep/*`) | Green in CI | -| Operator-level validation on a real GPU | **Done — no behavioural difference**, see [Testing](#testing) | -| Linux/GCC build (`--use_webgpu static_plugin`) | **Green in CI** (run `33030694040`) after the `array-bounds` fix — see [GCC](#gcc-and-warnings-as-errors) | -| Minimal build | Not yet exercised — see [Open items](#open-items) | -| Emscripten / ORT Web | **Builds and links**, plugin registration verified in the binary — see [ORT Web](#emscripten-and-ort-web). Not yet run in a browser | -| PR | Not opened yet | - -## Branch - -- Branch: `edgchen1/webgpu_static_plugin`, pushed to `git@github.com:microsoft/onnxruntime.git` (not a fork — - this matters, because the self-hosted 1ES CI pools are not available to forks). -- Forked from `main` at `fab4691797560a22092d9bbe8cec540b5413724b`. -- The earlier commits on the branch (`dff9cc42bf` .. `800ad881ef`) are the WebGPU extraction planning documents - and are unrelated to this workstream's code. The code work starts at `2754674229`. - -| Commit | Summary | -| --- | --- | -| `2754674229` | Support registering statically linked plugin EPs, and build WebGPU EP that way | -| `32775c305a` | Add CI jobs for the WebGPU static plugin EP build | -| `49855f6ef7` | Fix GCC maybe-uninitialized errors in the EP adapter attribute accessors | -| `d47f00a0a6` | Update dynamic plugin EP test config parsing tests | -| `1980e901ec` | Record CI coverage in the static plugin EP registration design doc | -| `754e025ddc` | Add a work log for the static plugin EP branch | -| `a362107df9` | Merge `origin/main` (52 commits behind at the time; merged, not rebased) | -| `35e6f65d52` | Record real-GPU validation results in the work log | -| `a110948b1d` | Record GPU validation and the plugin-path test coverage gap in the design doc | -| `8d87c11972` | Avoid GCC 14 `-Warray-bounds` false positive in `EmitGateActivationExpr` — the batch-3 CI fix | -| `ef68884a95` | Record the complete GCC 14 warning inventory and the local reproduction recipe | - -Pushed through `ef68884a95`. The push from `754e025ddc` was a fast-forward; no history has been rewritten on -this branch. - -## What the change consists of - -Read the design doc first. The short version, keyed to the decisions: - -- **D1** — A link-time registry in ORT core (`onnxruntime/core/session/plugin_ep/ep_static_plugins.{h,cc}`), - hand-written and `#if`-guarded, mirroring `EpLibraryInternal::CreateInternalEps`. Code generation from CMake - and a new public C API were both considered and rejected. -- **D2** — `ORT_PLUGIN_EP_ENTRY_POINT_PREFIX` in `onnxruntime/core/providers/webgpu/ep/api.cc`. The shared - library build must keep exporting **unprefixed** `CreateEpFactories` / `ReleaseEpFactory`, because - `EpLibraryPlugin::Load` looks those up by name. The static build emits `WebGpu_`-prefixed symbols instead. -- **D3** — No new ABI hook for process-global teardown. `ShutdownProtobufLibrary()` is compiled out unless - `ORT_PLUGIN_EP_OWNS_PROCESS_GLOBALS`. The legacy WebGPU cleanup block in `ort_env.cc` must stay for the - default internal-EP build. -- **D4** — CMake option `onnxruntime_WEBGPU_STATIC_PLUGIN` plus the derived - `onnxruntime_WEBGPU_LINKED_INTO_HOST`; surfaced as `--use_webgpu static_plugin`. -- **D5** — `ep_library_path` became optional in the dynamic plugin EP test infrastructure's - `InitializationConfig`, so a statically registered EP can be selected without a path to load. -- **D6** — Publish-then-register in `ort_env.cc`, and `OrtEnv::m_` became a `std::recursive_mutex`. Registration - reenters the environment (`GetSupportedDevices()` → `Api().ep.GetEnvConfigEntries()` → - `OrtEnv::TryGetInstance()`), which self-deadlocked with a plain mutex. -- **D7** — `ORT_PLUGIN_EP_STATICALLY_LINKED` disables the manual C++ API init in the shared - `include/onnxruntime/ep/api.h`. Do not try to force `ORT_API_MANUAL_INIT` on for everyone: MSVC's - `#pragma detect_mismatch` requires whole-binary agreement and doing so produced roughly 50 `LNK2038` errors. - -Files touched outside `docs/`: - -``` -.github/workflows/linux_webgpu.yml -.github/workflows/windows_webgpu.yml -cmake/CMakeLists.txt -cmake/onnxruntime.cmake -cmake/onnxruntime_providers_webgpu.cmake -cmake/onnxruntime_python.cmake -cmake/onnxruntime_unittests.cmake -include/onnxruntime/core/session/environment.h -include/onnxruntime/ep/adapter/op_kernel_info.h -include/onnxruntime/ep/api.h -onnxruntime/core/providers/webgpu/ep/api.cc -onnxruntime/core/providers/webgpu/math/{gemm,softmax,top_k}.h -onnxruntime/core/providers/webgpu/tensor/cast.h -onnxruntime/core/session/environment.cc -onnxruntime/core/session/ort_env.{h,cc} -onnxruntime/core/session/plugin_ep/ep_library_plugin.cc -onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.{h,cc} (new) -onnxruntime/core/session/plugin_ep/ep_library_static_plugin.{h,cc} (new) -onnxruntime/core/session/plugin_ep/ep_static_plugins.{h,cc} (new) -onnxruntime/test/framework/dynamic_plugin_ep_test.cc -onnxruntime/test/framework/inference_session_test.cc -onnxruntime/test/unittest_main/test_main.cc -onnxruntime/test/unittest_util/test_dynamic_plugin_ep.{h,cc} -tools/ci_build/{build.py,build_args.py} -``` - -## Building - -Three WebGPU configurations exist and it is worth being able to build at least two of them, because several -bugs in this work only appeared in one of them: - -``` -python tools/ci_build/build.py ... --use_webgpu # internal EP (the default, pre-existing path) -python tools/ci_build/build.py ... --use_webgpu shared_lib # plugin EP in its own shared library -python tools/ci_build/build.py ... --use_webgpu static_plugin # plugin EP linked into the host <-- the new one -``` - -`--use_webgpu static_plugin` maps to `-Donnxruntime_USE_EP_API_ADAPTERS=ON --Donnxruntime_WEBGPU_STATIC_PLUGIN=ON` (see `tools/ci_build/build.py`, the `--use_webgpu` handling). Both -options default to `OFF`. - -`onnxruntime_USE_EP_API_ADAPTERS=ON` is **incompatible** with `onnxruntime_BUILD_DAWN_SHARED_LIBRARY=ON` -(`cmake/CMakeLists.txt` raises a `FATAL_ERROR`). The internal-EP WebGPU CI job sets the latter; the plugin jobs -must not. - -### Windows gotchas - -These cost real time on the original machine and are likely to recur: - -- **`MAX_PATH`.** Building WebGPU under a long path such as `\build\Windows` makes Dawn's DXC build fail - with `MSB3491`. Build into a short directory instead — `C:\b` and `C:\b2` were used for the `static_plugin` - and `shared_lib` configurations respectively. With the Visual Studio generator the binaries land in - `\\\`. - - Refinement from a later machine: this is really an **MSBuild** problem. With `--cmake_generator Ninja` and - the registry's `LongPathsEnabled=1`, building at `D:\source\onnxruntime_4\build\sp` worked fine, so an - in-repo build directory is usable if you avoid MSBuild. Note that with Ninja the binaries land in - `\\`, one level, not two. -- **CMake may not be on `PATH`.** On a Visual Studio install it is at - `C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin`. -- Incremental rebuild of just the EP is much faster than a full build: - `cmake --build --config Debug --target onnxruntime_providers_webgpu --parallel`. - -### Verifying the D2 export contract - -For the **shared library** configuration, confirm the exported entry points are still unprefixed, since -`EpLibraryPlugin::Load` resolves them by name: - -``` -dumpbin /exports \\\onnxruntime_providers_webgpu.dll -``` - -Expect exactly `CreateEpFactories` and `ReleaseEpFactory`. The `static_plugin` build should instead produce -`WebGpu_`-prefixed symbols and no DLL. - -## Testing - -Relevant tests: - -- `onnxruntime_test_all --gtest_filter=InferenceSessionTests.WebGpu*` — covers registration, device - enumeration, and session creation end to end through the public V2 API. -- `onnxruntime_provider_test` — WebGPU EP kernel tests. - -The original development machine was a **Hyper-V VM with no real GPU**. On such a host WebGPU only surfaces a -*virtual* `OrtEpDevice`, and any session that is not compile-only is rejected up front with -*"WebGPU EP was selected on a virtual GPU device…"*. So locally only these two pass: - -- `InferenceSessionTests.WebGpuVirtualDeviceCompileOnlyEndToEnd` -- `InferenceSessionTests.WebGpuVirtualDeviceRejectedWithoutCompileOnly` - -and `WebGpuCompileOnlySkipsFinalization` and `TestStrictShapeInference` fail. This was proven to be a property -of the host and not of static linking, by building `--use_webgpu shared_lib` on the same machine and observing -the identical pass/fail set. - -**Consequence for whoever picks this up:** operator execution cannot be validated on a GPU-less machine at all. -Two ways to get real coverage: - -1. **Use CI.** The new `webgpu_static_plugin_build_x64_RelWithDebInfo` job in `.github/workflows/windows_webgpu.yml` - runs on the `onnxruntime-github-Win2022-GPU-A10` pool, which is the only GPU-equipped WebGPU pool, and it - both builds and tests. It has passed. This is currently the primary source of operator-level truth. -2. **Run locally on a real GPU.** Nothing in the change is GPU-vendor specific; a normal Windows machine with a - D3D12 or Vulkan capable GPU is enough. - -### Results on a real GPU - -This has now been done, on a machine with an NVIDIA RTX 5060 Ti. Both configurations were built from the same -tree, RelWithDebInfo, Ninja, differing **only** in the EP path: - -``` -python tools\ci_build\build.py --config RelWithDebInfo --build_dir --skip_submodule_sync --parallel \ - --cmake_generator Ninja --use_webgpu [static_plugin] --update --build --skip_tests \ - --cmake_extra_defines onnxruntime_BUILD_UNIT_TESTS=ON \ - onnxruntime_ENABLE_DAWN_BACKEND_D3D12=1 onnxruntime_ENABLE_DAWN_BACKEND_VULKAN=1 -``` - -`onnxruntime_provider_test`, with `ONNXRUNTIME_TEST_GPU_DEVICE_ID=0`: - -| | static_plugin | internal EP | -| --- | --- | --- | -| Total | 5894 | 5938 | -| Passed | 5775 | 5803 | -| Failed | 0 | 0 | -| Skipped | 119 | 135 | - -**Across the 5894 tests common to both builds there were zero status differences.** This is the evidence that -routing the static build through the plugin path changes no operator behaviour. - -`onnxruntime_test_all`: 1965 passed on both, no differences. All three `InferenceSessionTests.WebGpu*` pass on -the `static_plugin` build, including `WebGpuCompileOnlySkipsFinalization` — confirming the earlier failure of -that test was a property of the GPU-less host, as suspected. - -Two categories of test legitimately differ between the builds; neither is a regression: - -- **44 tests exist only in the internal-EP build.** 16 are `DISABLED_` and never run. The other 28 are - white-box tests that include internal WebGPU EP headers (`WebGpuContextTest`, `ActivationCacheKeyTest`, - `MatMul2BitsWebGpu`, `HardSwish_WebGPU`); `cmake/onnxruntime_unittests.cmake` (the - `onnxruntime_USE_WEBGPU AND NOT onnxruntime_USE_EP_API_ADAPTERS` guard) excludes `test/providers/webgpu/*` - from adapter builds entirely. Of these, `HardSwish_WebGPU` and `MatMul2BitsWebGpu` (12 tests) are real - operator coverage that exists only on the internal path — a known coverage gap of the plugin path, worth - either documenting in the design doc or closing by porting them to EP-agnostic tests. -- **The `InferenceSessionTests.WebGpu*` virtual-device tests are deliberately paired, not shared.** The - internal build has `WebGpuEpFactoryVirtualDevice`, `WebGpuEpFactoryRejectsVirtualDeviceWithoutCompileOnly` - and `WebGpuCompileOnlyUsesNoOpAllocator`; the plugin build has `WebGpuVirtualDeviceCompileOnlyEndToEnd` and - `WebGpuVirtualDeviceRejectedWithoutCompileOnly`, which `inference_session_test.cc` documents in-source as the - plugin-build counterparts. Only `WebGpuCompileOnlyUsesNoOpAllocator` has no counterpart, because it asserts - an internal allocator *type* via `dynamic_cast`, which is inherently unobservable across the ABI boundary. - -One caveat for whoever repeats this: do **not** run the two `onnxruntime_test_all` binaries concurrently. -`PathValidationTest`'s symlink tests collide over a shared temp directory and produce a spurious, and -non-deterministic, mix of pass/fail/skip. Run sequentially: all 31 pass on both builds. - -There is also a `webgpu-local-testing` skill in `.agents/skills/` describing how to run WebGPU tests on Linux -without a GPU using Mesa lavapipe (software Vulkan). Note its stated limitation: any graph containing `MatMul` -crashes lavapipe, so it only validates host-side enforcement and shape logic. - -### Not covered by any test yet - -Re-registration — register, run, unregister, re-register — is untested and is not reachable from the existing -test binaries, because `OrtEnv` is a refcounted process singleton that the unit test main pins for the lifetime -of the process. - -## CI - -No pre-existing leg built the `static_plugin` configuration, since both CMake options default to `OFF`. Every -WebGPU job was either internal EP (`--use_webgpu`) or shared library plugin (`--use_webgpu shared_lib`). Two -jobs were added: - -- `webgpu_static_plugin_build_x64_RelWithDebInfo` in `.github/workflows/windows_webgpu.yml` — builds **and - tests** on the A10 GPU pool. It deliberately omits `onnxruntime_BUILD_DAWN_SHARED_LIBRARY=ON` (incompatible, - see above) and deliberately drops the `--disable_rtti` / `--enable_lto` flags that the shared library job - carries, since those describe how the plugin ships as a DLL and here they would apply to all of ORT. -- `build-linux-webgpu-static-plugin-x64-release` in `.github/workflows/linux_webgpu.yml` — build only, mirroring - its sibling job. This is the only **GCC** coverage of the EP adapters, and the only coverage of the - `cmake/onnxruntime_python.cmake` change, which is why `--build_wheel` is kept on it. - -`linux_webgpu.yml` is build-only by design; its test job is commented out with a pre-existing "currently -failing" TODO. - -### Running CI on this branch - -The workflows only trigger automatically on `push`/`pull_request` for `main`, `rel-*`, and -`plugin-ep-webgpu/rel-*`, so **pushing this branch triggers nothing**. Dispatch manually: - -``` -gh workflow run windows_webgpu.yml --repo microsoft/onnxruntime --ref edgchen1/webgpu_static_plugin -gh workflow run linux_webgpu.yml --repo microsoft/onnxruntime --ref edgchen1/webgpu_static_plugin -gh workflow run windows_cuda_plugin.yml --repo microsoft/onnxruntime --ref edgchen1/webgpu_static_plugin -``` - -`gh workflow run --ref ` reads the workflow file **from that ref**, so jobs added on the branch do run -even though they are not on `main` yet. This was confirmed empirically. - -`windows_cuda_plugin.yml` matters because the CUDA plugin EP shares `include/onnxruntime/ep/*` with WebGPU, so -it is the regression check for the D5 and D7 changes. - -### Retrieving logs - -`gh run view --log-failed` and `gh run view --job --log` both returned **zero bytes** for these runs. Use -the REST API instead: - -``` -gh api repos/microsoft/onnxruntime/actions/runs//jobs --jq '.jobs[] | select(.conclusion=="failure") | "\(.id) \(.name)"' -gh api repos/microsoft/onnxruntime/actions/jobs//logs > log.txt -``` - -When grepping the result, search for `FAILED:` **case-sensitively**. A case-insensitive search for "failed" or -a search for `error:` drowns in benign CMake and Dawn feature-probe output such as -`-- Performing Test ... - Failed`. The same trap applies to local build logs: `grep -c "error:"` matches Dawn's -own configure line `-- DAWN Werror: OFF`, which looks alarming in a summary count and is nothing. - -### Run history - -| Run | Workflow | Result | -| --- | --- | --- | -| 32914301463 | ONNX Runtime WebGPU Builds | Success, all 6 jobs including the new one | -| 32914304408 | Linux WebGPU CI | Failure — GCC `maybe-uninitialized`, batch 1 | -| 32914307098 | CUDA Plugin Windows CI | Failure — `ParseInitializationConfigRejectsMissingRequiredFields` | -| 32996643911 | Linux WebGPU CI | Failure — GCC `maybe-uninitialized`, batch 2 | -| 32996646394 | CUDA Plugin Windows CI | Success | -| 33009901327 | Linux WebGPU CI | Failure — GCC `array-bounds`, batch 3. Internal-EP job on the same run passed. | -| 33030694040 | Linux WebGPU CI | **Success** — both jobs. Confirms the `array-bounds` fix; first green GCC run on this branch. | -| 33030695742 | ONNX Runtime WebGPU Builds | Dispatched on `ef68884a95` — re-run after the merge from `main` | -| 33030697208 | CUDA Plugin Windows CI | Dispatched on `ef68884a95` — re-run after the merge from `main` | - -## GCC and warnings as errors - -This was the only substantial problem the new CI surfaced, and it will bite again, so it is worth understanding -rather than pattern-matching. - -The EP adapter types are defined **entirely inline in headers**. The in-tree `OpKernelInfo` hides its -accessors behind out-of-line definitions in a `.cc`, so GCC cannot see their failure and exception paths. With -the adapters, it can — and it then warns about values that are only ever read once the call has succeeded. The -warning is a false positive, but the build uses `-Werror`. - -This had never been observed before simply because the EP-adapters path had never been compiled with GCC: the -existing Linux WebGPU CI builds the internal EP, and the shared library plugin builds are Windows/MSVC-only. -The CUDA plugin EP hit the identical problem earlier and solved it with a target-scoped suppression in -`cmake/onnxruntime_providers_cuda_plugin.cmake`. - -Two batches of failures appeared, because ninja stops early and only reports the targets that were already in -flight — **do not assume the first batch is the complete list**. A third batch, of a different warning class, -appeared later still. The lesson is the general one: **each CI round trip reveals only one batch**, so prefer a -local GCC build (or a throwaway dispatch with `--compile_no_warning_as_error`) to enumerate all of them at once. - -- Batch 1, fixed at the source in `49855f6ef7`: uninitialized locals that are passed by pointer and read back, - and the temporaries inside `GetAttrOrDefault` / `GetAttr` in `include/onnxruntime/ep/adapter/op_kernel_info.h`. - Value-initializing them (`T tmp{}`) is correct and worth keeping on its own merits. -- Batch 2: false positives deep inside `absl::InlinedVector`, for `TensorShapeVector` locals in - `tensor/pad.cc` and `tensor/unsqueeze.h` that are always properly constructed. These cannot be fixed at the - source. Handled by applying the same `-Wno-maybe-uninitialized` suppression the CUDA plugin EP uses, scoped - to the `onnxruntime_providers_webgpu` target, GCC only, and only when `onnxruntime_USE_EP_API_ADAPTERS` is on - so that non-adapter builds stay strict. -- Batch 3, in run `33009901327`, **fixed at the source**: `-Werror=array-bounds`, a *different* warning class, at - `onnxruntime/contrib_ops/webgpu/quantization/matmul_nbits_mlp.cc:47` in `EmitGateActivationExpr`: - - ``` - ‘void* __builtin_memcpy(...)’ forming offset [32, 48] is out of the bounds [0, 32] - of object ‘’ with type ‘std::__cxx11::basic_string’ - ``` - - GCC 14 at `-O3`, inlining the `std::string{gate_var} + " * (one / (one + exp(-" + std::string{gate_var} + ")))"` - temporary-move concat chain through `std::operator+(basic_string&&, const char*)`. A false positive. Note that - the batch-2 `-Wno-maybe-uninitialized` suppression **is** working — it is present on the failing command line - and the batch-2 files compiled clean. The file is pre-existing (added by PR #28280, `88ca23fcd5`), not authored - by this branch; the adapters PCH pulls in `include/onnxruntime/ep/adapters.h`, which changes inlining and is - why only the adapters config trips it. Fixed by building the expression with ORT's `MakeString` instead of a - `std::string` concat chain — a single-site source fix, preferred over extending the target-scoped suppression - with `-Wno-array-bounds`, which would have blinded the whole provider to a genuinely useful warning. - -### The complete GCC 14 warning inventory - -The batch-at-a-time discovery loop was finally closed by enumerating **every** warning in one pass: a full -container build with `--compile_no_warning_as_error`, then grouping the log by warning class. The result: - -| Warning class | Count | Where | -| --- | --- | --- | -| `-Wattributes` | 276 | Dawn generated code (`dawn_proc.cpp`) | -| `-Wunused-parameter` | 55 | ONNX (`onnx/defs/**`) | -| `-Wdangling-pointer=` | 3 | Dawn / Tint | -| `-Wchanges-meaning` | 3 | Dawn (`SystemEvent.h`) | -| `-Wstringop-overflow=` | 2 | flatbuffers (`reflection.cpp`) | -| `-Warray-bounds=` | **1** | **ORT — `matmul_nbits_mlp.cc:47`** | -| `-Wstringop-overread` | 1 | telemetry (`sqlite3_retail.c`) | - -**Exactly one warning in the whole build is attributable to ORT's own source**, and it is the batch-3 failure. -Everything else is third-party, in targets that do not compile with `-Werror`. So there is no batch 4: fixing -this one site is expected to turn the leg green. - -### Reproducing the CI compiler locally - -Worth the setup cost — it turns a ~28 minute CI round trip per batch into a local loop, and it is the only way -to enumerate all batches at once. - -The CI base image (`onnxruntimebuildcache.azurecr.io/.../cpu_x64_almalinux8_gcc14`) requires ACR auth and is -**not pullable**. An equivalent image is easy to build, and reproduces the compiler exactly -(GCC 14.2.1 20250110, Red Hat) — `almalinux:8` + `epel-release` + `gcc-toolset-14` + `python3.11`, with -`cmake ninja packaging numpy setuptools wheel` from pip. Notes: - -- **Python 3.11 is required, not the distro's 3.9** — `tools/ci_build/build_args.py` uses a `match` statement - (3.10+). CI itself uses `/opt/python/cp310-cp310/bin`. -- `--allow_running_as_root` is required inside the container. -- **No X11 dev packages are needed**, because `cmake/external/onnxruntime_external_deps.cmake` sets - `DAWN_USE_X11 OFF` unconditionally. This is what keeps the image minimal. -- Clone into the WSL native filesystem, not `/mnt/d` — building over the 9p mount is crippling. Only - `cmake/external/onnx` and `cmake/external/libprotobuf-mutator` need initialising for a native Linux build. - -To re-check a single file without paying for a whole build, override `CXX_FLAGS` on the generated makefile -rule, which lets you promote warnings back to errors against an existing build tree: - -```bash -FLAGS=$(sed -n 's/^CXX_FLAGS = //p' CMakeFiles/onnxruntime_providers_webgpu.dir/flags.make) -make -f CMakeFiles/onnxruntime_providers_webgpu.dir/build.make \ - CMakeFiles/onnxruntime_providers_webgpu.dir/src//matmul_nbits_mlp.cc.o \ - CXX_FLAGS="$FLAGS -Werror" -``` - -Always run the **negative control** — rebuild the unfixed source through the same harness and confirm it still -errors. Otherwise a harness that silently fails to exercise the warning looks exactly like a successful fix. - -### Verification of the fix - -Both halves were checked, because each one alone is inconclusive: - -1. **Single-object recheck.** With the fix, `matmul_nbits_mlp.cc.o` compiles clean under `-Werror`. Reverting the - file and rebuilding through the identical harness reproduces the exact CI error, which is what proves the - harness was actually exercising the warning. -2. **Full build with `-Werror` on**, i.e. the real CI configuration with `--compile_no_warning_as_error` dropped: - completes to 100%, links `onnxruntime_provider_test`, zero compiler errors. Before trusting this, confirm - `-Werror` really is on the ORT targets — `sed -n 's/^CXX_FLAGS = //p' CMakeFiles/.dir/flags.make` - should show it for `onnxruntime_providers_webgpu`, `onnxruntime_providers` and `onnxruntime_session` — and - confirm the fixed file appears in the build log, so it was recompiled rather than served from a stale object. - -The 334 warnings that remain in that build are all third-party, in targets that do not use `-Werror`. - -## Emscripten and ORT Web - -This is the motivating use case for the whole change, so it was built locally before opening the PR. **It builds -and links, and the plugin registration is verifiably present in the output.** - -Note what ORT Web's own CI does *not* cover today: the WebGPU WASM jobs in -`.github/workflows/linux-wasm-ci-build-and-test-workflow.yml` pass a bare `--use_webgpu`, and bare `--use_webgpu` -defaults to `static_lib` (`tools/ci_build/build_args.py`, `const="static_lib"`). So every existing WASM job still -builds the **internal** EP. Nothing in CI exercises WASM + `static_plugin`; this branch only makes the -configuration available, it does not switch ORT Web over to it. - -The CMake side was already written with Emscripten in mind — `cmake/onnxruntime_providers_webgpu.cmake` rejects -`shared_lib` there with `"WebGPU EP shared library build is not supported on Emscripten. Please use -'--use_webgpu static_plugin'."` - -### Building it - -Same container as the GCC 14 reproduction (see [above](#reproducing-the-ci-compiler-locally)), plus: - -- `git submodule update --init --recursive cmake/external/emsdk` — not needed for a native Linux build, required - here. `build.py` installs and activates the toolchain itself. -- **`node` must be on `PATH`.** `cmake/external/onnxruntime_webassembly.cmake` includes `node_helper.cmake`, - which does a hard `find_program(NODE_EXECUTABLE ...)` and fails configure with - `Could not find NODE_EXECUTABLE using the following names: node.exe, node`. CI gets node from - `actions/setup-node`; the AlmaLinux image has none. emsdk bundles one — point `PATH` at - `cmake/external/emsdk/node//bin`. - -``` -docker run --rm -v /home/edch/ort:/src -v /home/edch/ortwasm:/build -w /src ort-gcc14:local \ - bash -lc "export PATH=/src/cmake/external/emsdk/node/22.16.0_64bit/bin:\$PATH && \ - python3.11 tools/ci_build/build.py --build_dir /build --config Release --skip_submodule_sync --parallel \ - --allow_running_as_root --build_wasm --enable_wasm_simd --enable_wasm_threads \ - --enable_wasm_api_exception_catching --use_webgpu static_plugin --use_webnn \ - --target onnxruntime_webassembly --skip_tests" -``` - -Result: configure clean, zero compile failures, links `ort-wasm-simd-threaded.asyncify.mjs` + -`.wasm` (~32 MB). Roughly 35 minutes including the emsdk toolchain download and sysroot generation. - -### Verifying the plugin path actually linked - -A successful build alone proves little here, because the registration is behind -`#if defined(USE_WEBGPU) && defined(ORT_WEBGPU_STATIC_PLUGIN)` in -`onnxruntime/core/session/plugin_ep/ep_static_plugins.cc`. If that define failed to reach ORT core, the file -would still compile — `CreateStaticPluginEpLibraries()` would just return an empty vector, and the build would -be just as green while registering nothing. - -The decisive check is that ORT core holds an **undefined** reference to the prefixed entry points, using the -emsdk LLVM tools: - -``` -NM=cmake/external/emsdk/upstream/bin/llvm-nm -$NM /Release/libonnxruntime_session.a | grep WebGpu_ # expect: U WebGpu_CreateEpFactories, U WebGpu_ReleaseEpFactory -$NM /Release/libonnxruntime_providers_webgpu.a | grep WebGpu_ # expect: T WebGpu_CreateEpFactories, T WebGpu_ReleaseEpFactory -``` - -Observed exactly that: `U` in core, `T` in the provider, and the link resolved them. That closes the loop — -the `#if` was live, the entry point prefixing worked, and `EpLibraryStaticPlugin` is in the binary. - -### Still open - -The binary has not been **run**. `--skip_tests` was used, and exercising it needs a browser with WebGPU, which -the container does not have. So this validates build, link and registration wiring on Emscripten, not runtime -behaviour. - -## Open items - -1. ~~**Fix the Linux leg.**~~ **Done.** The batch-3 `array-bounds` break is fixed at the source, and a local - GCC 14 build confirmed it was the *only* ORT-owned warning in the whole build (see - [The complete GCC 14 warning inventory](#the-complete-gcc-14-warning-inventory)). CI run `33030694040` - is green on both jobs. -2. ~~**Operator-level validation on a real GPU.**~~ **Done** — see [Results on a real GPU](#results-on-a-real-gpu). - No behavioural difference. Remaining sub-item: decide whether to close the 12-test - `HardSwish_WebGPU` / `MatMul2BitsWebGpu` coverage gap on the plugin path, or document it as known. -3. ~~**Minimal build.**~~ **Resolved.** Exactly one pipeline combines an extended-minimal build with WebGPU — - `webgpu_minimal_build_edge_build_x64_RelWithDebInfo` in `.github/workflows/windows_webgpu.yml`, which passes a - bare `--use_webgpu` (internal EP) and is green, so the `#if !defined(ORT_MINIMAL_BUILD)` guard in `ort_env.cc` - is already exercised in CI. No ORT Web or wasm job uses a minimal build at all. The remaining hole — minimal - plus `static_plugin` linking the EP in without registering it — is now a configure-time `FATAL_ERROR` in - `cmake/CMakeLists.txt`, verified to fire on that combination and not on a non-minimal `static_plugin` - configure. Lifting the restriction properly is follow-up work, recorded in D4 of the design doc. -4. ~~**Emscripten / ORT Web.**~~ **Builds and links**, with the registration verified in the binary — see - [Emscripten and ORT Web](#emscripten-and-ort-web). Not yet run in a browser. -5. **Follow-up cleanup** listed at the end of the design doc, including removing the WebGPU special cases that - only exist while it is still an "internal" EP. -6. **Open the PR.** - -## Environment notes - -Specific to the original machine, but the class of problem generalises. - -- The working copy `C:\source\ort-webgpu-static-plugin` is a **linked git worktree** of `C:\source\onnxruntime`. - `.git` is therefore a *file*, not a directory. Do not write scratch files into it. -- Linting is `lintrunner`, driven by the `ort-lint` skill. `lintrunner -a` can report "ok No lint issues" and - "Successfully applied all patches" in the same run, so always re-check `git status` afterwards. -- The repository's own pre-commit hook lives in `.githooks/pre-commit` (this is not the `pre-commit` framework) - and is enabled with `git config core.hooksPath .githooks`. It **silently does nothing if `lintrunner` is not - on `PATH`**, so activate the virtual environment before committing or the commit goes through unlinted. diff --git a/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md b/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md index fa043d3dab4f0..638b696382ab5 100644 --- a/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md +++ b/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md @@ -262,7 +262,7 @@ Package signing, provenance, and release controls should meet the same requireme ORT should consume WebGPU source using the standard mechanism used for comparable third-party source dependencies. The dependency inventory should compare existing ORT mechanisms before selecting the exact implementation. -### Cross-repository version policy +## Cross-repository version policy Each repository pins the other, so the pins must be arranged so that the dependency does not become circular. Only one lane floats, and it never blocks a merge: @@ -381,3 +381,6 @@ The provider lives in its own repository: - Which copied ORT helpers need independent namespaces or API cleanup before transfer? - How should reduced-operator configuration be represented as an external provider input? - Which platforms and architectures are required for the first independent release? +- Which ORT-standard dependency mechanism should consume the external WebGPU source for WebAssembly builds? +- What compatibility window should the WebGPU EP promise across ORT releases? +- May the WebGPU build-against ORT version reference a pre-release, to shorten the wait for a newly added EP API? diff --git a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md index 39c64437a9294..1296c5e6f6a99 100644 --- a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md +++ b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md @@ -367,7 +367,9 @@ The report should make newly unsupported or newly excluded cases easy to detect `GetCapability()`? - Should contrib-op cases ship in the default kit or a separate ORT-extension bundle? - Which runner artifacts should be included in each ORT package and release channel? -- How should large datasets be deduplicated and downloaded? +- How should tensor data shared across cases be deduplicated? +- How should large datasets be versioned and distributed? - What compatibility promise applies to case, provider-profile, and report schema versions? - How should randomized or generated inputs remain deterministic and reproducible? -- What is the minimum representative operator set needed before using the suite as an extraction prerequisite? +- What is the minimum representative set of operators and data types needed before using the suite as an extraction + prerequisite? diff --git a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md index b263b3bf1e1c9..9b4a63051b955 100644 --- a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md +++ b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md @@ -15,6 +15,15 @@ existing coverage is accounted for and continues to run against the isolated or The detailed general-purpose conformance design is in [Execution Provider Operator Conformance Suite](ep_operator_conformance_design.md). +## Terminology + +| Term | Meaning | +| --- | --- | +| Conformance suite | The portable operator cases together with their execution and result semantics | +| Conformance kit | The versioned artifact published with an ORT release, containing cases, schemas, and a runner | +| Case archive | The platform-neutral conformance cases and schemas within a conformance kit | +| Provider profile | An EP's declared support surface, options, exclusions, and comparison overrides, supplied when running the conformance suite | + ## Desired end state - Every current WebGPU-related test has a stable owner and destination. @@ -207,8 +216,6 @@ tested behavior is an extraction blocker. ## Open questions - What exact current test set defines the extraction regression baseline? -- Which operators and data types form the minimum representative conformance set? - What additional reporting is needed for cases that intentionally permit partial graph assignment? - Which browser tests can consume the same portable cases without changing semantics? -- How should large test datasets be versioned and distributed? - Which test generators should remain in ORT, move, or be copied? diff --git a/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md b/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md index e98952daeb12b..b5ab642b94ed8 100644 --- a/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md +++ b/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md @@ -39,10 +39,6 @@ provider into its WebAssembly module. | --- | --- | | Plugin EP API | The public ORT API used to implement a plugin EP: `OrtApi`, `OrtEpApi`, `OrtEpFactory`, and `OrtEp` | | Staging root | `plugin-ep-webgpu/`, the in-tree directory the provider is consolidated into before the move | -| Conformance suite | The portable operator cases together with their execution and result semantics | -| Conformance kit | The versioned artifact published with an ORT release, containing cases, schemas, and a runner | -| Case archive | The platform-neutral conformance cases and schemas within a conformance kit | -| Provider profile | An EP's declared support surface, options, exclusions, and comparison overrides, supplied when running the conformance suite | ## Workstreams @@ -55,9 +51,6 @@ The effort is divided into four workstreams: | `test-conformance` | [Test ownership and operator conformance](test_ownership_and_conformance/test_ownership_and_conformance_workstream.md) | Existing coverage is preserved, every test has an owner, and portable conformance coverage protects the external provider | | `node-migration` | [Node plugin migration](node_plugin_migration/node_plugin_migration_workstream.md) | Existing bundled Node WebGPU support is replaced by an explicitly consumable plugin without regressing current users | -The detailed static plugin EP registration design is in -[Static Plugin EP Registration](plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md). - The detailed reusable conformance-suite design is in [Execution Provider Operator Conformance Suite](test_ownership_and_conformance/ep_operator_conformance_design.md). @@ -79,7 +72,26 @@ implementation-specific even though the tests are not, so a passing default-bund provider being extracted. This is handled in [Test ownership and operator conformance](test_ownership_and_conformance/test_ownership_and_conformance_workstream.md). -## Workstream dependencies +## Consumer dispositions + +Every consumer that receives WebGPU today needs a recorded disposition before the built-in implementation is removed +from ORT. Platform and architecture details remain to be inventoried in the individual workstreams. + +| Consumer or package | Disposition | Extraction requirement | +| --- | --- | --- | +| `onnxruntime-web` | Consume a pinned external WebGPU source revision through static plugin registration | Required | +| Python WebGPU plugin package | Move the existing optional plugin package and release pipeline | Required | +| WebGPU NuGet plugin package | Move the existing optional plugin package and release pipeline | Required | +| Node WebGPU support | Provide a tested replacement for the WebGPU implementation currently bundled in `onnxruntime-node`; final package naming is open | Required before bundled support is removed | +| `onnxruntime-webgpu` on PyPI | Publication has already stopped; do not resurrect it or convert it into a plugin-dependent package | Confirm whether any other retired package needs the same treatment | + +Hosts that do not ship WebGPU support today are outside this table. Adding one is a separate decision and is not an +extraction prerequisite. + +The table should be updated as the current package and platform inventory is completed. Dropping a consumer requires +an explicit compatibility decision rather than silently removing support. + +## Sequencing Most work proceeds concurrently, but one sequence determines the end date: @@ -108,90 +120,17 @@ The remaining convergence points are not on the critical path: - The Node workstream consumes generic plugin loading from ORT and native WebGPU artifacts from the external provider. - The final repository copy depends on `plugin-ep-webgpu/` being independently buildable. -## Consumer dispositions - -Every consumer that receives WebGPU today needs a recorded disposition before the built-in implementation is removed -from ORT. Platform and architecture details remain to be inventoried in the individual workstreams. - -| Consumer or package | Disposition | Extraction requirement | -| --- | --- | --- | -| `onnxruntime-web` | Consume a pinned external WebGPU source revision through static plugin registration | Required | -| Python WebGPU plugin package | Move the existing optional plugin package and release pipeline | Required | -| WebGPU NuGet plugin package | Move the existing optional plugin package and release pipeline | Required | -| Node WebGPU support | Provide a tested replacement for the WebGPU implementation currently bundled in `onnxruntime-node`; final package naming is open | Required before bundled support is removed | -| `onnxruntime-webgpu` on PyPI | Publication has already stopped; do not resurrect it or convert it into a plugin-dependent package | Confirm whether any other retired package needs the same treatment | - -Hosts that do not ship WebGPU support today are outside this table. Adding one is a separate decision and is not an -extraction prerequisite. - -The table should be updated as the current package and platform inventory is completed. Dropping a consumer requires -an explicit compatibility decision rather than silently removing support. - -## Integrated delivery stages - -### Stage 1: Establish baselines - -- Inventory private dependencies, build inputs, packages, consumers, platforms, and tests. -- Record every current test's owner and intended destination. -- Establish baselines for correctness, package installation, WebAssembly size, and inference latency. The later size - and performance gates are measured against these baselines. -- Confirm the consumer dispositions and platform coverage. - -### Stage 2: Execute workstreams in parallel - -The critical path runs through this stage: static plugin registration, then code isolation and the standalone build. -The remaining items proceed alongside it. - -- Add generic static plugin registration and run WebGPU through the plugin path. -- Replace the kernel-authoring foundation and consolidate provider-owned sources and build inputs under - `plugin-ep-webgpu/`, so the staging root is adapter-free and independently buildable. -- Preserve current tests while implementing the conformance MVP. -- Scaffold the external repository and CI against the standalone build contract. -- Design and implement the Node plugin-loading and package transition. - -### Stage 3: Prove extraction readiness - -The source move is ready when: - -- Static and dynamic forms execute the same provider implementation through the public plugin EP API. -- Process-global initialization and teardown in the static plugin setup work correctly. -- `plugin-ep-webgpu/` builds from a clean copy without private ORT implementation headers or libraries. -- Every existing test is classified and current tested behavior remains blocking. -- WebGPU-specific tests run from the isolated provider tree. -- A representative conformance set detects output failures, unsupported regressions, and CPU fallback. -- Existing Python and NuGet plugin packages can be produced from the isolated tree. -- Node has an approved package transition and a tested replacement path. -- `onnxruntime-web` passes browser integration, reduced-build, binary-size, and performance checks through static plugin - registration. - -### Stage 4: Move and switch consumers - -- Import the isolated provider code and selected history into the new repository. -- Publish versioned native plugin artifacts and immutable source artifacts for static consumers. -- Move Python and NuGet plugin releases to the WebGPU repository. -- Pin the WebGPU revision used by `onnxruntime-web`. -- Execute the approved Node package transition, including any package retirement it implies. -- Remove obsolete WebGPU implementation and packaging code from ORT. - ## Success criteria -- WebGPU has one runtime boundary with ORT: the public plugin EP API. -- Static and shared builds execute the same provider implementation and operator tests. -- `plugin-ep-webgpu/` can be copied and built outside the ORT source tree. +These are the outcomes that show the whole effort is complete: + +- Static and shared builds execute the same provider implementation, through the public plugin EP API, against the + same operator tests. +- The ORT repository retains provider integration shims only, and no longer contains WebGPU EP implementation, build, + or packaging inputs. - The external repository owns WebGPU code, dependencies, tests, packages, and releases. - ORT updates its pinned WebGPU revision through a routine dependency update. - `onnxruntime-web` preserves supported functionality and accepted size and performance characteristics. - Native ORT packages remain usable without installing WebGPU. - Existing Node WebGPU users have a documented and tested migration path. - Compatibility failures produce clear build-time, registration-time, or package-time diagnostics. - -## Cross-workstream decisions still open - -- What public API gaps remain after private dependencies are classified? -- What is the stable browser boundary for JavaScript `GPUDevice` and `GPUBuffer` objects? -- Which ORT-standard dependency mechanism should consume the external WebGPU source for WebAssembly builds? -- What platform and architecture matrix is required for the first independent release? -- What compatibility window should the WebGPU EP promise across ORT releases? -- Should the Node transition keep the `onnxruntime-node` name or introduce a core-only package such as - `onnxruntime-node-core`? -- May the WebGPU build-against ORT version reference a pre-release, to shorten the wait for a newly added EP API? From 020942424fc52f270e8abbd22024efd1bf83bcdf Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Tue, 1 Sep 2026 21:30:57 -0700 Subject: [PATCH 035/107] docs: fix inaccurate claims in WebGPU EP extraction overview Replace 'package-time' with 'install-time' in the compatibility diagnostics criterion. No other doc in the set uses 'package-time', and the failure point it meant is a package manager refusing an incompatible ORT and plugin combination at install. The three points are now ordered as a user reaches them: build, install, register. Correct the convergence points list, which claimed its items were not on the critical path. One of them restated the code-isolation to source-transfer edge, which is the critical path; that now sits in the preceding paragraph. Each remaining bullet states what it actually gates. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .../webgpu_ep_extraction.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md b/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md index b5ab642b94ed8..c60d46d968983 100644 --- a/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md +++ b/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md @@ -109,16 +109,19 @@ Static plugin registration is a prerequisite for isolation, not merely an enable moves provider sources into the staging root rather than copying them, so once isolation completes there is exactly one WebGPU implementation and it is adapter-free. The non-plugin static build that `onnxruntime-web` ships from today must already be served through static plugin registration before the adapter can be removed. Relocating the sources -is not itself blocked; removing the adapter is. +is not itself blocked; removing the adapter is. Source transfer then waits on the staging root being independently +buildable. -The remaining convergence points are not on the critical path: +The other convergence points between workstreams: - Provider isolation identifies private dependencies. A dependency becomes public API work only when an existing public API cannot express a necessary, stable runtime interaction and the proposed addition meets the high bar for - a permanent plugin EP API. -- Test classification determines which tests move with the provider and which remain in ORT. -- The Node workstream consumes generic plugin loading from ORT and native WebGPU artifacts from the external provider. -- The final repository copy depends on `plugin-ep-webgpu/` being independently buildable. + a permanent plugin EP API. Such an addition has to ship in an ORT release before the provider can build against it, + so it can extend isolation. +- Test classification determines which tests move with the provider and which remain in ORT. Source transfer waits on + it. +- The Node workstream consumes generic plugin loading from ORT and native WebGPU artifacts from the external + provider. It does not gate source transfer, but bundled Node WebGPU cannot be removed until it lands. ## Success criteria @@ -133,4 +136,4 @@ These are the outcomes that show the whole effort is complete: - `onnxruntime-web` preserves supported functionality and accepted size and performance characteristics. - Native ORT packages remain usable without installing WebGPU. - Existing Node WebGPU users have a documented and tested migration path. -- Compatibility failures produce clear build-time, registration-time, or package-time diagnostics. +- Compatibility failures produce clear build-time or registration-time diagnostics. From 18a2b8346e9bed490ad579e2b34866b92545011c Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Wed, 2 Sep 2026 13:54:42 -0700 Subject: [PATCH 036/107] minor updates --- .../plugin_boundary_and_web_integration_workstream.md | 4 ++-- .../provider_isolation_and_repository_migration_workstream.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md index f243fbc9c7da4..69b9b1c94235c 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md @@ -90,7 +90,7 @@ Parity work includes: - Error and diagnostic behavior. - Process, environment, factory, and session lifetime behavior. -The direct in-tree WebGPU provider path remains only as a temporary comparison baseline. +The WebGPU non-plugin path remains only as a temporary comparison baseline. ## Plugin API gap closure @@ -140,7 +140,7 @@ The bridge should not expose unrelated ORT private implementation details. plugin-path change lands, since the size and latency completion criteria are relative to them. 2. **Static registration core:** implement and contract-test generic static factory registration. 3. **Global lifetime contract:** inventory process-global state and implement safe ownership and teardown rules. -4. **Emscripten prototype:** compile the adapter path statically and run a small model. +4. **Emscripten prototype:** compile the plugin path statically and run a small model. 5. **Gap inventory triage:** convert private-dependency findings into public API or provider-owned actions. 6. **Browser bridge:** specify and prototype object and lifetime exchange. 7. **Parity and retirement:** run the existing suite through the plugin path and remove the legacy path. diff --git a/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md b/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md index 638b696382ab5..df054339cca3e 100644 --- a/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md +++ b/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md @@ -18,7 +18,7 @@ The WebGPU repository owns: - `OrtEpFactory` and `OrtEp` provider implementation. - WebGPU kernels, contrib kernels, runtime, and device infrastructure. -- WebGPU-owned support code copied or replaced from ORT. +- WebGPU-owned support code copied from ORT. - Dawn selection, patches, and build configuration. - WGSL templates, generators, and generated-source policy. - WebGPU-specific unit, integration, package, and regression tests, and browser tests of provider behavior. @@ -38,7 +38,7 @@ rather than moved. The WebNN code in that directory is unaffected by this work. Expand the existing `plugin-ep-webgpu/` directory into the in-tree staging root. It already owns Python and NuGet plugin packaging, version metadata, and release documentation. Add provider source, build, dependency, and test -ownership until the directory mirrors the intended external repository: +ownership until the directory mirrors the intended external repository. An example directory structure: ```text plugin-ep-webgpu/ From 551433fb7a62f4b4d377aed8159f89557904b0a8 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Wed, 2 Sep 2026 15:49:14 -0700 Subject: [PATCH 037/107] docs: address PR feedback on conformance results and CI relocation Simplify conformance result semantics to PASS, FAIL, and SKIP with a documented reason. A crash, a hang, or the absence of a compatible device ends the run instead of producing per-case results, so 'exactly one result per case' now holds for every result the runner can actually write. Let the conformance design doc own the runner, schemas, and result semantics. The test-conformance workstream now states only what the extraction requires from the suite, instead of restating the contract. Cases carry their own expected outputs, so a CPU reference run is not an extraction requirement; the design doc records that CPU runs exist to validate the cases themselves. Note that workflow files stay under the repository-root .github/workflows, since GitHub discovers them only there, and only the scripts, actions, and templates they call move to the staging root. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- ...ion_and_repository_migration_workstream.md | 4 +- .../ep_operator_conformance_design.md | 41 +++++++++------- ...st_ownership_and_conformance_workstream.md | 49 +++++-------------- 3 files changed, 37 insertions(+), 57 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md b/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md index df054339cca3e..beb65a472a437 100644 --- a/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md +++ b/docs/design/webgpu_ep_extraction/provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md @@ -163,7 +163,9 @@ validates its pinned WebGPU revision as part of ORT Web release gating. Python and NuGet plugin packaging sources already live under `plugin-ep-webgpu/`. The plugin build, test, and packaging pipeline definitions are still outside the staging root, split between `tools/ci_build/github/azure-pipelines/` and `.github/workflows/`. This workstream relocates those and rewires them -to invoke the standalone build, but does not redesign the packaging scripts themselves. +to invoke the standalone build, but does not redesign the packaging scripts themselves. Workflow files under +`.github/workflows/` are an exception: GitHub discovers them only at the repository root, so they stay there and only +the scripts, actions, and templates they call move. The pipelines invoke `tools/ci_build/build.py --use_webgpu shared_lib` and consume artifacts from ORT's build output locations, so they break the moment the standalone build replaces the in-tree one. The rewiring therefore lands with diff --git a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md index 1296c5e6f6a99..14daab0373fbc 100644 --- a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md +++ b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/ep_operator_conformance_design.md @@ -38,7 +38,7 @@ consume it as a stable release interface. - Support dynamically loaded and statically linked plugin EPs with the same cases and result rules. - Support native and browser/Wasm runners without changing case meaning. - Publish a versioned conformance kit associated with an ORT release. -- Make unsupported cases, exclusions, and tolerance overrides explicit and reportable. +- Make skipped cases and tolerance overrides explicit and reportable. - Allow existing `OpTester` and `ModelTester` cases to migrate incrementally. ## Non-goals @@ -57,7 +57,7 @@ The design separates case authorship, case distribution, and execution: 2. **Case generator** validates definitions and emits portable serialized models and datasets. 3. **Conformance kit** packages generated cases, schemas, documentation, and a native runner for an ORT release. 4. **Runner** registers an EP, executes selected cases, enforces fallback rules, and writes a structured report. -5. **Provider profile** declares the EP's expected support surface, options, exclusions, and narrow comparison +5. **Provider profile** declares the EP's expected support surface, options, skips, and narrow comparison overrides. The portable contract is the case format and execution/result semantics. A particular runner implementation is not @@ -104,12 +104,11 @@ Generated `.onnx` models and large tensor datasets do not normally need to be ch generate them into a conformance-kit archive. A model should be checked in only when its exact serialized form is part of the test or generation is not reasonably deterministic. -Provider repositories own their profiles, exclusions, and implementation-specific tests. For example: +Provider repositories own their profiles and implementation-specific tests. For example: ```text webgpu-ep/tests/conformance/ provider-profile.json - exclusions.json ``` ## Case representation @@ -166,7 +165,8 @@ For an ordinary operator conformance case, the runner should: 1. Load or register the requested plugin EP and select a device. 2. Apply the requested EP options. -3. Create a session with CPU fallback disabled, unless the CPU EP is itself the target. +3. Create a session with CPU fallback disabled through `session.disable_cpu_ep_fallback`, unless the CPU EP is + itself the target. 4. Load the generated or packaged model. 5. Require the target EP to accept the nodes specified by the case. 6. Run every input dataset. @@ -185,17 +185,17 @@ should be classified separately from single-operator correctness cases. Each selected case should produce exactly one result: - `PASS`: The target EP executed the required graph and all outputs matched. -- `UNSUPPORTED`: The EP did not claim a case outside its declared support profile. -- `FAIL`: Output mismatch, crash, timeout, unexpected rejection, or unexpected fallback. -- `EXCLUDED`: A provider exclusion matched the case and supplied a documented reason. -- `NOT_RUN`: The environment could not execute the case, such as when no compatible device was available. +- `FAIL`: Output mismatch, or the target EP did not execute the required graph. +- `SKIP`: The provider profile says the case is not expected to run, with a documented reason. -`UNSUPPORTED` is not automatically a passing result. If the provider profile says the case is supported, rejection -is a failure. `NOT_RUN` should make a required CI lane incomplete rather than successful. +A skip is legitimate only when the provider profile says so. If the profile does not skip a case and the target EP +will not run it, that is a failure. -An exclusion should identify a stable case ID, a reason, and preferably a tracking issue. Broad wildcard skip lists +A skip should identify a stable case ID, a reason, and preferably a tracking issue. Broad wildcard skip lists should be discouraged because they obscure coverage loss. +If no compatible device is available, the run fails rather than reporting every case as skipped. + ## Comparison semantics The case owns the default comparison policy. The policy may specify: @@ -218,7 +218,7 @@ may contain: - Device selection and EP options. - Supported domains, opsets, data types, and optional features. - Case tags to include or exclude from a particular environment. -- Documented exclusions. +- Documented skips, each with a reason. - Narrow comparison overrides. The runner uses the profile to distinguish an expected lack of support from a regression in the provider's declared @@ -287,6 +287,11 @@ target_link_libraries( This executable should consume the same case archive and produce the same report as the dynamic runner. Static and dynamic linkage must not create separate conformance definitions. +The shared runner core must therefore stay on the public API boundary. The prebuilt dynamic executable could +technically link ORT-private test libraries, since ORT builds and distributes it as a self-contained binary, but the +same core has to be consumable by an EP repository building the static form. Requiring private ORT headers or +libraries there would couple that repository to ORT's source layout and private C++ ABI. + ## WebAssembly and browser usage `onnxruntime-web` cannot use the native dynamic-plugin executable. A JavaScript or browser runner should load the same @@ -294,7 +299,7 @@ portable cases, invoke the statically registered WebGPU plugin through ORT Web, schema and outcome rules. Browser-specific scheduling, test sharding, and artifact loading are host concerns. They must not change the meaning -of `PASS`, `FAIL`, `UNSUPPORTED`, `EXCLUDED`, or `NOT_RUN`. +of `PASS`, `FAIL`, or `SKIP`. ## Report format @@ -306,10 +311,10 @@ Reports should be machine-readable and include enough provenance to reproduce a - Dynamic or static registration mode. - Device and relevant environment information. - Provider profile hash. -- Per-case result, duration, and diagnostics. +- Per-case result, skip reason, duration, and diagnostics. - Summary counts by result, domain, operator, opset, and data type. -The report should make newly unsupported or newly excluded cases easy to detect in CI. +The report should make newly skipped cases easy to detect in CI. ## Migration approach @@ -319,7 +324,7 @@ The report should make newly unsupported or newly excluded cases easy to detect - Implement a public-API-only native runner for dynamic plugin EPs. - Enforce CPU-fallback prevention. - Convert a small representative set of ONNX and contrib operators. -- Run those cases against CPU and at least one plugin EP. +- Run those cases against CPU to validate the cases themselves, and against at least one plugin EP. ### Phase 2: Connect existing test infrastructure @@ -348,7 +353,7 @@ The report should make newly unsupported or newly excluded cases easy to detect - The same case detects and fails unexpected CPU fallback. - Dynamic and static forms of one plugin produce equivalent results. - An external EP repository can run a conformance kit without an ORT source checkout. -- Results distinguish failures, unsupported cases, exclusions, and infrastructure failures. +- Results distinguish failures from skips and record why each case was skipped. - Existing provider coverage can be mapped to stable conformance case IDs without an all-at-once migration. ## Open questions diff --git a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md index 9b4a63051b955..896c6d278dbb4 100644 --- a/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md +++ b/docs/design/webgpu_ep_extraction/test_ownership_and_conformance/test_ownership_and_conformance_workstream.md @@ -22,7 +22,7 @@ The detailed general-purpose conformance design is in | Conformance suite | The portable operator cases together with their execution and result semantics | | Conformance kit | The versioned artifact published with an ORT release, containing cases, schemas, and a runner | | Case archive | The platform-neutral conformance cases and schemas within a conformance kit | -| Provider profile | An EP's declared support surface, options, exclusions, and comparison overrides, supplied when running the conformance suite | +| Provider profile | An EP's declared support surface, options, skips, and comparison overrides, supplied when running the conformance suite | ## Desired end state @@ -110,49 +110,22 @@ classification and verifies that replacement coverage is equivalent. The initial conformance milestone should be deliberately bounded. It needs to prove the contract required for safe extraction, not complete migration of ORT's operator test suite. -The MVP should provide: - -- A shared conformance-runner core for case loading, execution, comparison, and reporting. -- A prebuilt native executable that uses the core and loads dynamic plugin EP libraries. -- A small SDK or CMake target that uses the same core and lets an external EP link its static factory registration, - if a native static-linkage runner is adopted. -- CPU fallback prevention or reliable assignment verification. -- Case, provider-profile, and report schemas. -- Structured `PASS`, `UNSUPPORTED`, `FAIL`, `EXCLUDED`, and `NOT_RUN` results, with semantics defined in - [Execution Provider Operator Conformance Suite](ep_operator_conformance_design.md). -- A representative set of ONNX and contrib cases spanning important data types, shapes, options, and failure modes. -- Execution against CPU as a reference and WebGPU in shared and static forms. -- A versioned artifact usable without an ORT source checkout. - -The representative set should be chosen from the current WebGPU support surface and include enough diversity to -exercise capability discovery, model loading, execution, output comparison, unsupported behavior, and fallback -detection. - -The dynamic and static runners should be thin registration frontends over the same runner core so that linkage mode -does not change case interpretation or result semantics. - -The shared runner core must interact with ORT through released public APIs. The dynamic executable could technically -link ORT-private test libraries because ORT builds and distributes it as a self-contained binary. The same core, -however, must also be consumable by an external EP repository to build a statically linked runner. Requiring private -ORT headers or libraries there would couple the external repository to ORT's source layout and private C++ ABI. -Keeping the shared core on the public API boundary allows ORT to distribute one implementation for both frontends and -lets an external EP build the static form against a released ORT package without an ORT source checkout. +[Execution Provider Operator Conformance Suite](ep_operator_conformance_design.md) owns the runner, schemas, and +result semantics. This extraction requires that suite to deliver: + +- A representative set of cases chosen from the current WebGPU support surface, with enough diversity to exercise + capability discovery, model loading, execution, output comparison, profile-declared skips, and fallback detection. +- Execution of that set against WebGPU in both shared and static forms, checked against the expected outputs the + cases carry. +- A versioned artifact the external provider repository can run without an ORT source checkout. ## Coverage continuity rules - A test may be deleted only after its replacement is blocking in the required CI lanes. - A moved test must protect the same behavior and platforms unless a reduction is explicitly approved. -- An unsupported result is a failure when the provider profile declares support. -- `NOT_RUN` does not satisfy a required lane. -- Exclusions require stable case IDs, reasons, and preferably tracking issues. -- Provider-wide tolerance inflation is not an acceptable migration shortcut. - Forking a shared ORT helper transfers the behavior it implements to the provider. Where that behavior was covered only incidentally by tests of another consumer, the inventory entry must record whether existing coverage follows the fork or new provider-side coverage is required. -- Static and dynamic forms should share case definitions and outcome semantics. -- For cases requiring full target-EP assignment, use the existing `session.disable_cpu_ep_fallback` session option. - Cases intentionally allowing partial assignment need explicit assignment requirements and may require additional - reporting. - Keep the inventory current while extraction is in progress. New WebGPU tests must be classified when added, and CI should detect test files or registrations missing from the inventory where practical. @@ -164,7 +137,7 @@ lets an external EP build the static form against a released ORT package without 4. **Conformance schemas and runner:** implement the public execution and reporting contract. 5. **Representative case conversion:** convert a bounded extraction-gate set. 6. **External CI integration:** run existing and conformance coverage against clean provider artifacts. -7. **Coverage reporting:** detect newly unsupported, excluded, or unexecuted cases. +7. **Coverage reporting:** detect newly skipped cases. Inventory, runner design, and provider-specific relocation can proceed concurrently. Static runner validation depends on the `plugin-boundary` workstream's static registration facility. @@ -181,7 +154,7 @@ on the `plugin-boundary` workstream's static registration facility. - Supplies the test ownership map and required destinations. - Requires standalone provider artifacts and CI environments. -- Moves provider profiles, exclusions, and implementation tests with the provider. +- Moves provider profiles and implementation tests with the provider. ### Node plugin migration From ec4b772e6070e00fd7bc0a9c51f141e511700234 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Wed, 2 Sep 2026 20:38:35 -0700 Subject: [PATCH 038/107] Record four-side web benchmark: b876d290cd has no effect on the wasm build Settles the open question of whether the shared-allocator zero-init deferral (b876d290cd) does anything on web/wasm, using a four-side experiment: main_lib (merge-base), lib, plugin_nofix (branch head with the commit reverted) and plugin. Each side was built from its own worktree and its own build directory, with no reuse and no --skip_submodule_sync; all four .wasm hashes are distinct. The primary endpoint is the deterministic GPUQueue.submit count rather than timing. Reverting the commit does not change it: 300 dispatches over 299 intermediates still batch into exactly 20 submits with or without the deferral. The per-intermediate-buffer submit the commit describes does not occur on wasm. Timing agrees and is noise - the sign of plugin - plugin_nofix flips between the two models. Also records the lib vs main_lib null, which isolates the plugin-path cost from anything else on the branch. That control is unusually clean because the two sides' bundles came out byte-identical, leaving the .wasm as the only variable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../static_plugin_ep_work_log.md | 137 +++++++++++++++++- 1 file changed, 130 insertions(+), 7 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md index 04309e6f32feb..64b0b5b38a96a 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md @@ -480,11 +480,131 @@ $NM /Release/libonnxruntime_providers_webgpu.a | grep WebGpu_ # expect: Observed exactly that: `U` in core, `T` in the provider, and the link resolved them. That closes the loop — the `#if` was live, the entry point prefixing worked, and `EpLibraryStaticPlugin` is in the binary. -### Still open - -The binary has not been **run**. `--skip_tests` was used, and exercising it needs a browser with WebGPU, which -the container does not have. So this validates build, link and registration wiring on Emscripten, not runtime -behaviour. +### Running it in a browser + +The wasm binary has now been **run**, on a real GPU, in Edge. A dedicated harness lives in `build/bench_web/` +(untracked scratch, not part of the repo): + +- `make_side.ps1 -Side ` builds one **self-consistent "side"**: it copies that side's + `ort-wasm-simd-threaded.asyncify.{wasm,mjs}` into `js/web/dist`, checks out the matching + `js/web/lib/wasm/session-options.ts`, runs `npm run build -- --bundle-mode=perf --webgpu-ep`, and snapshots the + result with a `MANIFEST.txt` recording git SHA, source provenance and SHA-256 of every artifact. +- `serve.js` serves the snapshots on port 8099; `bench.js` is the page. +- `run_bench.ps1 -Sides @(...)` drives an interleaved (ABBA / cyclic-rotation) run across N sides. + +Two properties of the harness matter more than the timings: + +- **`--webgpu-ep` is mandatory and fails silently if omitted.** Without it `js/web/script/build.ts` sets + `DISABLE_WEBGPU=true` and the session quietly runs on **JSEP** instead, producing a plausible-looking but + completely irrelevant number. +- **A bundle and a wasm are a matched pair.** `build.ts` has an esbuild `onLoad` hook that *inlines the `.mjs` + loader into the bundle*, so a bundle from one build silently carries another build's loader. Sides must never + be mixed by hand. + +### Verifying the plugin path is actually taken at runtime + +Linking is not execution, and the failure mode here is silent, so the runtime path was pinned down with +controls rather than by reading code: + +| control | bundle | wasm | result | +| --- | --- | --- | --- | +| `plugin` | plugin | `static_plugin` | runs, correct output | +| `mismatch` | plugin | `static_lib` | **runs, correct output** — the silent trap | +| `mismatch2` | lib | `static_plugin` | fails: "WebGPU execution provider is not supported in this build" | +| bogus EP name | patched | `static_plugin` | fails: "No execution provider device is registered for `…BOGUS`" | + +The bogus-name failure goes on to list the registered devices — `CPUExecutionProvider` and +`WebGpuExecutionProvider`. That control is the load-bearing one: it proves the V2 name lookup is real and +enforced, so a *successful* `plugin` run necessarily resolved a genuinely registered plugin EP device. + +The page also patches `GPUQueue.prototype.submit` to count GPU submissions. That counter has been **perfectly +deterministic** across every run, which makes it a far better endpoint than any timing statistic. + +### Measured cost of the plugin path on the web build + +Comparing `static_lib` against `static_plugin` wasm built from the *same* commit — the two `build.py` argument +dumps are identical except `use_webgpu='static_lib'` vs `'static_plugin'` — over 6-8 interleaved rounds per +model, taking the median of each round's minimum: + +| model | nodes | `static_lib` | `static_plugin` | delta | predicted @ 4.76 us/node | residual | +| --- | --- | --- | --- | --- | --- | --- | +| `bench_compute` | 16 | 5.00 ms | 5.00 ms | 0.00 ms | 0.08 ms | -0.08 ms | +| `bench_dispatch_150` | 150 | 4.30 ms | 5.10 ms | +0.80 ms | 0.71 ms | +0.09 ms | +| `bench_dispatch` | 300 | 7.90 ms | 9.40 ms | +1.50 ms | 1.43 ms | +0.07 ms | +| `bench_dispatch_600` | 600 | 13.80 ms | 16.60 ms | +2.80 ms | 2.86 ms | -0.06 ms | + +The plugin side was slower in **26 of 26** paired rounds. A single through-origin constant of **~4.8 us of CPU +per kernel node** fits all four points to within one 0.1 ms timer tick, and **GPU submit counts were +byte-identical on every side, model and round** (3 per run for compute; 11/20/39 for 150/300/600). So this is +pure per-kernel CPU dispatch overhead, not extra queue work. The 16-node null is not evidence of no cost — the +predicted 0.08 ms simply sits below the timer floor. + +A three-way run isolated where the cost is **not**: the `mismatch` side (plugin bundle, `static_lib` wasm) came +out level with `lib` (median delta -0.20 ms, a 3/6 sign split), while `plugin` was +1.60 ms in 6/6 rounds. +The JS-side `_OrtAppendExecutionProviderV2` call and EP-name lookup are therefore free; the cost is inside the +`static_plugin` wasm's per-kernel C++ path. Where exactly is still unknown — 4.8 us is roughly 10k cycles, +far too much for an indirect call, so it points at real per-kernel work. An `--enable_wasm_profiling` build +would give named frames. + +Timer resolution is 0.1 ms: Edge coarsens `performance.now()` without cross-origin isolation. Enabling +COOP/COEP would give ~5 us resolution and was deliberately *not* turned on mid-experiment, to keep all numbers +comparable. + +### Correction: what the shared-allocator commit does and does not explain + +An earlier reading of these results as "refuting" the design doc's +36% figure was **wrong**, and the mistake is +worth recording. Commit `b876d290cd` ("Defer zero-initialize submission in the WebGPU plugin EP shared +allocator") is an **ancestor** of the benchmarked branch head, so every `plugin` number above is already +*plugin-with-the-fix*. Parity on the compute-bound model with identical submit counts is therefore +**consistent with that fix working**, not evidence against the original measurement. + +A second retraction: the claim that the earlier *native* A/B was "null by construction" because +`cmake/onnxruntime_providers_webgpu.cmake` excludes the `ep/` folder from non-plugin builds was also wrong. The +build logs show both native sides were `use_webgpu='shared_lib'`, which **does** compile `ep/factory.cc`. That +null was genuine. + +What remains genuinely untested is whether the deferral matters on **web/wasm** at all. That was settled by +a four-side experiment — `main_lib` (merge-base `2fee06a0c1`), `lib`, `plugin_nofix` (branch head with +`b876d290cd` reverted, commit `a6be215b31`) and `plugin` — each built from its **own worktree and its own build +directory**, with no reuse and no `--skip_submodule_sync`. The **primary endpoint is the deterministic submit +count, not timing**: the commit's own mechanism claim is "a queue submit for every intermediate buffer allocated +mid-run", so `plugin_nofix` must show more submits per run than `plugin` if it does anything at all. +`bench_dispatch` has 299 intermediates against `bench_compute`'s 15, making it ~20x the detector. + +### Result: `b876d290cd` has no measurable effect on the web build + +The submit counter is perfectly deterministic — the same value in all 8 rounds of every side, both models: + +| model | dispatches/run | submits/run: `main_lib` | `lib` | `plugin_nofix` | `plugin` | +| --- | --- | --- | --- | --- | --- | +| `bench_compute` | 16 | 3 | 3 | 3 | 3 | +| `bench_dispatch` | 300 | 20 | 20 | 20 | 20 | + +**Reverting the commit changes nothing.** On `bench_dispatch`, 300 dispatches with 299 intermediates still batch +into exactly 20 submits whether the deferral is present or not. The mechanism the commit describes — one submit +per intermediate buffer — simply **does not occur on the wasm build**, so there is nothing there for it to defer. + +Timing agrees, and is noise: `plugin - plugin_nofix` med `dmin` is **+1.70 ms (6/8 rounds *slower*)** on +`bench_dispatch` but **-0.30 ms (3/8)** on `bench_compute`. The sign flips between models, which is the +signature of build-to-build variation (code layout / icache) rather than a real effect. Note the direction: the +side *with* the fix is if anything marginally slower on the dispatch-heavy model. + +Combined with the earlier genuine native A/B null, `b876d290cd` has **no demonstrated benefit on any platform +tested**, and no correctness change was identified. It should not be landed on its own merits. + +Two other contrasts fall out of the same run, on the `medmin` delta: + +| pair | `bench_compute` | `bench_dispatch` | reading | +| --- | --- | --- | --- | +| `lib - main_lib` | 0.00 ms (0.0%) | -0.10 ms (-0.8%) | the branch does **not** regress the static_lib path | +| `plugin - lib` | +0.30 ms (+5.8%) | +3.00 ms (+25.2%) | cost of the plugin path itself | +| `plugin - main_lib` | +0.30 ms (+5.8%) | +2.90 ms (+24.2%) | **total ship delta vs. `main`** | + +The `lib` vs `main_lib` null is the important control: it isolates the plugin-path cost from "anything else the +branch changed". The isolation is unusually clean here because the two sides' `ort.all.min.js` came out +**byte-identical** (SHA-256 `79391F71...`), the merge-base and `main` blobs of `session-options.ts` being the +same — so the *only* difference between those two sides is the `.wasm`. The same holds for +`plugin_nofix` vs `plugin` (bundle `AB2F004F...`). All four `.wasm` hashes were confirmed distinct. ## Open items @@ -502,8 +622,11 @@ behaviour. plus `static_plugin` linking the EP in without registering it — is now a configure-time `FATAL_ERROR` in `cmake/CMakeLists.txt`, verified to fire on that combination and not on a non-minimal `static_plugin` configure. Lifting the restriction properly is follow-up work, recorded in D4 of the design doc. -4. ~~**Emscripten / ORT Web.**~~ **Builds and links**, with the registration verified in the binary — see - [Emscripten and ORT Web](#emscripten-and-ort-web). Not yet run in a browser. +4. ~~**Emscripten / ORT Web.**~~ **Builds, links, and now runs in a browser** on a real GPU — see + [Running it in a browser](#running-it-in-a-browser). Registration is verified in the binary *and* at runtime. + Open sub-item: the plugin path costs **~4.8 us of CPU per kernel node** on the web build + ([measurement](#measured-cost-of-the-plugin-path-on-the-web-build)); the source of that cost is not yet + identified and needs an `--enable_wasm_profiling` build to attribute. 5. **Follow-up cleanup** listed at the end of the design doc, including removing the WebGPU special cases that only exist while it is still an "internal" EP. 6. **Open the PR.** From 1dd5a5567b007b93a275c3579c52fab8972a2c9f Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Wed, 2 Sep 2026 17:00:20 -0700 Subject: [PATCH 039/107] Revert "Defer zero-initialize submission in the WebGPU plugin EP shared allocator" This reverts commit b876d290cdc51506dbea35d46d3f92ff10e84024. --- .../core/providers/webgpu/ep/factory.cc | 20 +++++------ .../core/providers/webgpu/webgpu_context.h | 34 ------------------- .../webgpu/webgpu_execution_provider.cc | 11 ++---- .../webgpu/webgpu_execution_provider.h | 11 ++---- .../providers/webgpu/webgpu_context_test.cc | 30 ---------------- 5 files changed, 15 insertions(+), 91 deletions(-) diff --git a/onnxruntime/core/providers/webgpu/ep/factory.cc b/onnxruntime/core/providers/webgpu/ep/factory.cc index a4bf61f751ac0..3ecf8c2b36cc1 100644 --- a/onnxruntime/core/providers/webgpu/ep/factory.cc +++ b/onnxruntime/core/providers/webgpu/ep/factory.cc @@ -248,16 +248,16 @@ OrtStatus* ORT_API_CALL Factory::CreateAllocatorImpl( "Unsupported memory info for shared allocator."); } - // Defer the zero-initialize submission while a run is in flight so that the clear batches with the run's own - // dispatches. This allocator is not bound to an EP instance, so it reads the run state from the context. - auto create_allocator = [](const OrtMemoryInfo&) -> AllocatorPtr { - return std::make_shared( - []() -> const webgpu::BufferManager& { return WebGpuContextFactory::DefaultContext().BufferManager(); }, - false, - []() { return !WebGpuContextFactory::DefaultContext().IsRunActive(); }); - }; - - *allocator = new onnxruntime::ep::adapter::Allocator(memory_info, create_allocator); + *allocator = new onnxruntime::ep::adapter::Allocator(memory_info, + [](const OrtMemoryInfo&) -> AllocatorPtr { + return std::make_shared( + []() -> const webgpu::BufferManager& { + return WebGpuContextFactory::DefaultContext() + .BufferManager(); + }, + false, + []() { return true; }); + }); return nullptr; EXCEPTION_TO_RETURNED_STATUS_END } diff --git a/onnxruntime/core/providers/webgpu/webgpu_context.h b/onnxruntime/core/providers/webgpu/webgpu_context.h index 93efa3a80eee2..1790c85e7214a 100644 --- a/onnxruntime/core/providers/webgpu/webgpu_context.h +++ b/onnxruntime/core/providers/webgpu/webgpu_context.h @@ -3,9 +3,6 @@ #pragma once -#include -#include -#include #include #include #include @@ -308,32 +305,6 @@ class WebGpuContext final { void CollectProfilingData(); void EndProfiling(TimePoint, profiling::Events& events); - // Whether a Session::Run is currently in flight on this context. - // - // Buffer zero-initialization consults this to decide whether the clear must be submitted before - // returning: during a run the clear is encoded into the current command encoder and rides along with - // the run's own submission, which preserves dispatch batching. Outside a run there is no upcoming - // submission to piggyback on, so the clear is submitted immediately. - // - // This lives on the context (rather than only on WebGpuExecutionProvider) because the context owns the - // command encoder that batching depends on, and because allocators that are not bound to an EP instance - // must be able to observe it. - // - // A context is shared by every session that uses the same device. WebGpuExecutionProvider reports - // ConcurrentRunSupported() == false, so InferenceSession serializes runs, but that lock is per-session - // and does not serialize sessions sharing a context. This is therefore a count rather than a flag: - // with a flag, one session ending its run would report the context as idle while another session's - // run is still in flight. - bool IsRunActive() const { return active_run_count_.load() > 0; } - - // Must be called in pairs. WebGpuExecutionProvider::OnRunStart increments only on the path that - // returns success, because InferenceSession::Run arranges for OnRunEnd only when OnRunStart succeeded. - void IncrementActiveRunCount() { active_run_count_.fetch_add(1); } - void DecrementActiveRunCount() { - [[maybe_unused]] const auto previous_count = active_run_count_.fetch_sub(1); - assert(previous_count > 0); - } - // // Push error scope. // @@ -443,11 +414,6 @@ class WebGpuContext final { uint32_t num_pending_dispatches_ = 0; uint32_t max_num_pending_dispatches_ = 16; - // Number of Session::Run calls currently in flight on this context, across every session sharing it. - // Signed so that an unbalanced decrement fails toward "no run active" (an eager submit, which is - // always safe) instead of wrapping to a huge value that would look like a run is permanently active. - std::atomic active_run_count_{0}; - // Owns the active dispatch window and the unique pending builds referenced within that window. std::vector deferred_dispatches_; diff --git a/onnxruntime/core/providers/webgpu/webgpu_execution_provider.cc b/onnxruntime/core/providers/webgpu/webgpu_execution_provider.cc index 0659ac70a29cc..81cc974906cf3 100644 --- a/onnxruntime/core/providers/webgpu/webgpu_execution_provider.cc +++ b/onnxruntime/core/providers/webgpu/webgpu_execution_provider.cc @@ -854,19 +854,12 @@ Status WebGpuExecutionProvider::OnRunStart(const onnxruntime::RunOptions& run_op } } - // Incremented last so that it is only incremented on the path that returns success: InferenceSession::Run - // calls OnRunEnd only for providers whose OnRunStart succeeded, so an early return above must not have - // incremented the count. - context_.IncrementActiveRunCount(); + run_active_.store(true); return Status::OK(); } -bool WebGpuExecutionProvider::IsRunActive() const { - return context_.IsRunActive(); -} - Status WebGpuExecutionProvider::OnRunEnd(bool /* sync_stream */, const onnxruntime::RunOptions& run_options) { - context_.DecrementActiveRunCount(); + run_active_.store(false); // When capturing, flushing creates the replay-ready CapturedCommandInfo entries before // CaptureEnd() detaches their external storage. diff --git a/onnxruntime/core/providers/webgpu/webgpu_execution_provider.h b/onnxruntime/core/providers/webgpu/webgpu_execution_provider.h index e1cf340af51cb..fcf9db36c3b65 100644 --- a/onnxruntime/core/providers/webgpu/webgpu_execution_provider.h +++ b/onnxruntime/core/providers/webgpu/webgpu_execution_provider.h @@ -4,6 +4,7 @@ #pragma once +#include #include #include #include @@ -96,14 +97,7 @@ class WebGpuExecutionProvider : public IExecutionProvider { Status OnRunStart(const onnxruntime::RunOptions& run_options) override; Status OnRunEnd(bool sync_stream, const onnxruntime::RunOptions& run_options) override; - // Run state lives on the WebGpuContext because it owns the command encoder whose dispatch batching - // depends on it, and because allocators not bound to an EP instance (the plugin EP's shared allocator, - // created by the factory before any EP exists) must observe the same state. - // - // This reports context-wide activity: it is true while any session sharing this context has a run in - // flight, not just this one. That is what the batching decision depends on, since the command encoder - // being batched into is shared. - bool IsRunActive() const; + bool IsRunActive() const { return run_active_.load(); } // WebGPU EP reuses the Device ID as the key to get the WebGpuContext instance. int GetDeviceId() const override { return context_id_; } @@ -144,6 +138,7 @@ class WebGpuExecutionProvider : public IExecutionProvider { std::vector force_cpu_node_names_; bool enable_graph_capture_ = false; bool graph_buffer_mgr_active_ = false; + std::atomic run_active_{false}; bool enable_int64_ = false; uint32_t multi_rotary_cache_concat_offset_ = 0; uint32_t kv_cache_quantization_bits_ = 0; diff --git a/onnxruntime/test/providers/webgpu/webgpu_context_test.cc b/onnxruntime/test/providers/webgpu/webgpu_context_test.cc index 23da0d1d0bb48..70d2af300e816 100644 --- a/onnxruntime/test/providers/webgpu/webgpu_context_test.cc +++ b/onnxruntime/test/providers/webgpu/webgpu_context_test.cc @@ -222,36 +222,6 @@ TEST(WebGpuContextTest, WebGpuExecutionProviderTracksRunActivity) { EXPECT_FALSE(webgpu_ep->IsRunActive()); } -// Sessions that use the same device share one WebGpuContext, so run activity is tracked as a count. One -// session ending its run must not report the context as idle while another session's run is in flight. -TEST(WebGpuContextTest, RunActivityTracksEveryProviderSharingAContext) { - ConfigOptions options; - auto first_ep = WebGpuProviderFactoryCreator::Create(options)->CreateProvider(); - auto second_ep = WebGpuProviderFactoryCreator::Create(options)->CreateProvider(); - ASSERT_NE(first_ep, nullptr); - ASSERT_NE(second_ep, nullptr); - auto* first_webgpu_ep = static_cast(first_ep.get()); - auto* second_webgpu_ep = static_cast(second_ep.get()); - ASSERT_EQ(first_webgpu_ep->GetDeviceId(), second_webgpu_ep->GetDeviceId()); - RunOptions run_options; - - EXPECT_FALSE(first_webgpu_ep->IsRunActive()); - - ASSERT_STATUS_OK(first_webgpu_ep->OnRunStart(run_options)); - ASSERT_STATUS_OK(second_webgpu_ep->OnRunStart(run_options)); - // Both providers observe the shared context's activity, not just their own. - EXPECT_TRUE(first_webgpu_ep->IsRunActive()); - EXPECT_TRUE(second_webgpu_ep->IsRunActive()); - - ASSERT_STATUS_OK(first_webgpu_ep->OnRunEnd(false, run_options)); - EXPECT_TRUE(first_webgpu_ep->IsRunActive()); - EXPECT_TRUE(second_webgpu_ep->IsRunActive()); - - ASSERT_STATUS_OK(second_webgpu_ep->OnRunEnd(false, run_options)); - EXPECT_FALSE(first_webgpu_ep->IsRunActive()); - EXPECT_FALSE(second_webgpu_ep->IsRunActive()); -} - TEST(WebGpuContextTest, EnablesLazyClearResourceOnFirstUse) { #if defined(__wasm__) || defined(USE_EXTERNAL_DAWN) GTEST_SKIP() << "Dawn native toggle inspection is unavailable."; From de610ec8a51cf25ccfd995dfdf4be00246ec6866 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 3 Sep 2026 11:17:38 -0700 Subject: [PATCH 040/107] Record the disposition of b876d290cd: reverted and verified The four-side web benchmark showed no measurable effect, and the earlier native A/B was a timing null, so the commit was reverted rather than shipped. Records how the revert was verified: byte-identical to the pre-fix state, no dangling references to the removed WebGpuContext members, and all three reverted .cc files recompiled -- which needs both a static_plugin and an integrated build, since neither config compiles all three. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../static_plugin_ep_work_log.md | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md index 64b0b5b38a96a..884a8bf532c90 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md @@ -606,6 +606,33 @@ branch changed". The isolation is unusually clean here because the two sides' `o same — so the *only* difference between those two sides is the `.wasm`. The same holds for `plugin_nofix` vs `plugin` (bundle `AB2F004F...`). All four `.wasm` hashes were confirmed distinct. +### Disposition: `b876d290cd` reverted + +Given no measurable benefit on either the web or the native build, and no correctness issue that it fixes, the +commit was reverted rather than shipped. The revert restores the unconditional submit — the conservative +behaviour — so what is lost is a speculative optimization, not a guarantee. + +The revert was applied as a cherry-pick (a draft PR is already open, so history was not rewritten) and then +verified rather than assumed: + +- `git diff b876d290cd^ HEAD` over the five affected files is **empty**, and `git diff b876d290cd HEAD` is + **non-empty** — the tree is byte-identical to the pre-fix state. +- No dangling references to the removed `WebGpuContext` members remain (`IncrementActiveRunCount`, + `DecrementActiveRunCount`, `active_run_count_` — zero hits tree-wide). This matters because a clean + cherry-pick only proves the patch *applied*, not that later commits had not started using the deleted API. +- All three reverted `.cc` files recompile: `ep/factory.cc` and `webgpu_execution_provider.cc` in a + `static_plugin` build, and `webgpu_context_test.cc` in an integrated build. Note that no single config + compiles all three — `ep/factory.cc` is absent from the integrated build and the test is absent from the + plugin build — so both configs are needed for full coverage. + +The EP-level `WebGpuExecutionProvider::IsRunActive()` predates this commit and is unaffected; only the +context-level copy of the state was removed. + +Caveat on scope: this is "no measurable benefit in the scenarios we could construct", not proof of a universal +no-op. The evidence is strong for the web build, where the patched `GPUQueue.submit` counter is a *complete* +instrument for the claimed harm. It is weaker for native, where only timing was compared and submits were +never counted. + ## Open items 1. ~~**Fix the Linux leg.**~~ **Done.** The batch-3 `array-bounds` break is fixed at the source, and a local From ea79e2bbf7191a6ea6bc4a4952a4dbaf64f57402 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 3 Sep 2026 11:21:21 -0700 Subject: [PATCH 041/107] Remove the deferred-submission detour from the static plugin EP design doc The +36% compute-bound regression it reported does not reproduce, and the fix it proposed has been reverted for having no measurable effect. The design doc describes the design as it stands, so the detour is dropped rather than annotated; the work log keeps the record. Also drops the now-unsupported claim that sequential sampling understated a real regression -- the interleaving rationale stands on drift alone. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../static_plugin_ep_registration_design.md | 43 +------------------ .../static_plugin_ep_work_log.md | 4 ++ 2 files changed, 5 insertions(+), 42 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md index 41e2f6ed0b062..9a9c7642a4158 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -638,8 +638,7 @@ Three runs per model per build. **The two builds must be sampled interleaved** `plugin/dispatch, plugin/compute, lib/dispatch, lib/compute`, repeated — rather than all samples of one build followed by all samples of the other. This machine drifts: a build that has just finished, a warm browser, or a warm GPU shifts timings by more than the effect being measured, and sequential sampling aliases that drift onto -the build under test. The sequential methodology used for the first pass understated a real regression by nearly -an order of magnitude (see below). The table reports the median of the three per-run P50 values. +the build under test. The table reports the median of the three per-run P50 values. | Metric | `static_lib` | `static_plugin` | Delta | | --- | ---: | ---: | ---: | @@ -671,46 +670,6 @@ Two cautions on methodology: - Timings taken with `--webgpu.profiling.mode=default` are inflated by the timestamp queries and must never be compared against timings taken without it. -#### The shared allocator must defer zero-initialize submission during a run - -Interleaved sampling exposed a genuine, code-level regression in the plugin path that sequential sampling had -hidden: the compute-bound model ran at 9.90 ms against the control's 7.30 ms, **+36%**, with no overlap between -the two sample sets. That the dispatch-bound model — 300 nodes against the compute model's 16 — did *not* show a -proportional cost ruled out a fixed per-node or per-run overhead and pointed at something scaling with bytes -allocated rather than with node count. - -The cause was buffer zero-initialization. `BufferManager::Create` clears a buffer recycled from the cache, and -`GpuBufferAllocator` takes a `should_submit_zero_initialize` predicate deciding whether that clear is submitted -before `Create` returns. During `Session::Run` it must **not** be: the clear is encoded into the run's own command -encoder and rides along with the run's submission, preserving dispatch batching. Outside a run there is no -upcoming submission to piggyback on, so the clear must be submitted immediately. `WebGpuExecutionProvider` and the -plugin factory's per-EP allocator both passed `!IsRunActive()`, but the factory's **shared** allocator — the one -backing intermediate tensors — hardcoded `true`, forcing a queue submit for every intermediate buffer mid-run. -The compute model allocates sixteen 1 MB intermediates, so it paid sixteen forced submits per inference. - -The shared allocator is created by `CreateAllocatorImpl` before any EP instance exists, so it had no EP to ask — -which is why the predicate was stubbed out rather than wired up. The fix moves the run-active flag onto -`WebGpuContext`, which is both reachable from the allocator and the more correct owner: it holds -`current_command_encoder_` / `current_compute_pass_encoder_`, the very state that batching depends on. -`WebGpuExecutionProvider` sets it in `OnRunStart` / `OnRunEnd` and its `IsRunActive()` now delegates to the -context, so all three allocator sites read one source of truth. With the fix the compute-bound gap disappears into -noise, as the table above shows. - -This bug predates the ORT Web migration and affects any consumer of the WebGPU plugin EP, not just the web build; -it is a candidate to land separately from the static-plugin work. - -It was present in the pre-merge measurements too, which reported only +4% for this model. The reason is that the -cost is state-dependent rather than constant: `BufferManager::Create` only clears — and therefore only flushes — -a buffer it recycles from its cache, so the penalty depends on cache occupancy and appears as a wide, roughly -bimodal distribution rather than a fixed offset. Sequential sampling of the *same* defective binary produced -per-run P50s spanning 7.70 ms to 10.20 ms, a range that brackets the pre-merge figure. The merge itself did not -introduce or amplify the defect: of the three merged commits touching this path, #32272 only adds an early return -when an allocator is already registered (not reached with a single device), and #32269's subgroup-matrix MatMul is -gated on the F16 8x16x16 config, so the f32 compute-bound model never takes it. The lesson is about sample count -and interleaving on a high-variance metric, not about the merge. - - - > Should static factories be registered before environment creation or through environment construction options? Neither. They are registered by ORT core during `OrtEnv` creation, after the environment is constructed and diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md index 884a8bf532c90..607989c1b1dea 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md @@ -628,6 +628,10 @@ verified rather than assumed: The EP-level `WebGpuExecutionProvider::IsRunActive()` predates this commit and is unaffected; only the context-level copy of the state was removed. +The design doc's write-up of this detour — the `+36%` figure and the section proposing the deferred-submission +fix — has been removed from `static_plugin_ep_registration_design.md`, since that document describes the design +as it stands rather than the path taken to it. This work log keeps the record. + Caveat on scope: this is "no measurable benefit in the scenarios we could construct", not proof of a universal no-op. The evidence is strong for the web build, where the patched `GPUQueue.submit` counter is a *complete* instrument for the claimed harm. It is weaker for native, where only timing was compared and submits were From 2861a5f34b4745b8a35b8b690f84682092b7a03f Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 3 Sep 2026 11:35:16 -0700 Subject: [PATCH 042/107] Record the sync convention for the higher-level plan docs edgchen1/webgpu_ep_extraction_plan is the authoritative source for the overview and workstream docs, and deliberately excludes the static-plugin design doc and this work log. Records that higher-level edits made here must be upstreamed, that the merge is one-way, and that cross-reference- only edits are not upstreamable as-is. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../static_plugin_ep_work_log.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md index 607989c1b1dea..1225f08399f75 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md @@ -661,6 +661,23 @@ never counted. 5. **Follow-up cleanup** listed at the end of the design doc, including removing the WebGPU special cases that only exist while it is still an "internal" EP. 6. **Open the PR.** +7. **Keep the higher-level plan docs in sync with `edgchen1/webgpu_ep_extraction_plan`.** That branch is the + authoritative source for the overall extraction docs — the overview and the four workstream documents. It + deliberately does *not* carry `static_plugin_ep_registration_design.md` or this work log: it added them in + `a75b4c1294` and removed them again in `fdf2f53e0a`, so implementation-level docs live on this branch and the + new design docs are added when this branch lands. Until then, any change made here to a *higher-level* doc + must also be upstreamed to that branch. Two practical consequences: + - Do not merge this branch *into* the plan branch — that would restore the two files and undo `fdf2f53e0a`. + Merge in the other direction only, then upstream individual higher-level edits by hand. + - An edit that only adds a cross-reference to a doc that exists solely on this branch is *not* upstreamable + as-is; it would be a broken link there. Upstream the substantive part without the link, or defer it. + + Current delta against the plan branch, as of this entry: three hunks, all touching the static-plugin docs. + Two are pure cross-references and land with this branch. The third resolves a plan-level open question -- + static factories are registered by ORT core during `OrtEnv` creation, not before it and not through + environment construction options. That resolution is upstreamable on its own once the trailing link is + dropped; **decision: deferred until this branch lands**, at which point the link becomes valid and the whole + set can go across together. ## Environment notes From e75881f1a89758e80ce1f69922617ae0b6828f74 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 3 Sep 2026 12:33:03 -0700 Subject: [PATCH 043/107] Fix string-array attribute leak in the EP API OpKernelInfo adapter GetAttrsImpl() for std::vector leaked the allocator's array, and every string still in it, if reserve() or a std::string copy threw. Hold the array in a unique_ptr whose deleter frees each element. The deleter calls OrtAllocator::Free through the function pointer rather than Ort::AllocatorWithDefaultOptions::Free(), which throws on failure and so must not run in a destructor. Also drop the value-initialization rationale comment in GetAttrOrDefault(); GetAttr() right below already does the same thing without one. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../onnxruntime/ep/adapter/op_kernel_info.h | 24 ++++++++++++------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/include/onnxruntime/ep/adapter/op_kernel_info.h b/include/onnxruntime/ep/adapter/op_kernel_info.h index aa6c784c0a9c1..da2270f604fd1 100644 --- a/include/onnxruntime/ep/adapter/op_kernel_info.h +++ b/include/onnxruntime/ep/adapter/op_kernel_info.h @@ -158,8 +158,6 @@ struct OpKernelInfo { template [[nodiscard]] T GetAttrOrDefault(const std::string& name, const T& default_value) const { - // Value-initialize. These accessors are fully inline here, so the compiler can see the failure - // paths of GetAttr() and warns about a possibly uninitialized read without it. T tmp{}; return GetAttr(name, &tmp).IsOK() ? tmp : default_value; } @@ -275,17 +273,25 @@ struct OpKernelInfo { ORT_RETURN_IF_ERROR( ToStatus(Ort::GetApi().KernelInfoGetAttributeArray_string(info, name, allocator, &raw_values, &size))); + // The allocator owns both the array and every string in it, so release them on all exit paths: constructing the + // std::string copies below can throw. Free through the OrtAllocator function pointer rather than + // Ort::AllocatorWithDefaultOptions::Free(), which throws on failure and so must not run in this deleter. + OrtAllocator* raw_allocator = allocator; + auto free_raw_values = [raw_allocator, size](char** values_to_free) { + for (size_t i = 0; i < size; ++i) { + if (values_to_free[i] != nullptr) { + raw_allocator->Free(raw_allocator, values_to_free[i]); + } + } + raw_allocator->Free(raw_allocator, values_to_free); + }; + std::unique_ptr raw_values_guard{raw_values, std::move(free_raw_values)}; + std::vector values; values.reserve(size); for (size_t i = 0; i < size; ++i) { - if (raw_values[i] != nullptr) { - values.emplace_back(raw_values[i]); - allocator.Free(raw_values[i]); - } else { - values.emplace_back(); - } + values.emplace_back(raw_values[i] != nullptr ? raw_values[i] : ""); } - allocator.Free(raw_values); out.swap(values); return Status::OK(); } From 7e9fcdca995929880dc06518e2dbbbf60e32d909 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 3 Sep 2026 12:33:17 -0700 Subject: [PATCH 044/107] Collapse the plugin EP build macros and rename the manual-init switch The build defined three perfectly correlated macros: ORT_PLUGIN_EP_ENTRY_POINT_PREFIX and ORT_PLUGIN_EP_STATICALLY_LINKED on the static branch, ORT_PLUGIN_EP_OWNS_PROCESS_GLOBALS on the shared one. Drop the first and third; key everything off ORT_PLUGIN_EP_STATICALLY_LINKED and hardcode the WebGpu_ entry point prefix. Parameterizing only the provider side was fictional flexibility, since ORT core already spells WebGpu_CreateEpFactories out literally in ep_static_plugins.cc. Separately, rename the switch in the public header include/onnxruntime/ep/api.h to ORT_SKIP_API_MANUAL_INIT. It says nothing about linkage; it says whether OrtGetApiBase() is reachable in-process. Keep the negative polarity so that undefined continues to mean manual init on, which is the safe default for a separately built plugin. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- cmake/onnxruntime_providers_webgpu.cmake | 10 +++----- include/onnxruntime/ep/api.h | 13 ++++++----- onnxruntime/core/providers/webgpu/ep/api.cc | 23 ++++++++++--------- .../session/plugin_ep/ep_static_plugins.cc | 6 ++--- 4 files changed, 25 insertions(+), 27 deletions(-) diff --git a/cmake/onnxruntime_providers_webgpu.cmake b/cmake/onnxruntime_providers_webgpu.cmake index eb96d2ee33fa5..796f1d28a19ce 100644 --- a/cmake/onnxruntime_providers_webgpu.cmake +++ b/cmake/onnxruntime_providers_webgpu.cmake @@ -51,11 +51,11 @@ # Prefix the plugin EP entry points so that multiple statically linked plugin EPs can coexist in one binary. # ORT core declares the prefixed names in onnxruntime/core/session/plugin_ep/ep_static_plugins.cc. # - # ORT_PLUGIN_EP_STATICALLY_LINKED tells the shared plugin EP headers that OrtGetApiBase() is available in-process, + # ORT_SKIP_API_MANUAL_INIT tells the shared plugin EP headers that OrtGetApiBase() is available in-process, # so the C++ API must not be built with ORT_API_MANUAL_INIT (the rest of the binary is not). target_compile_definitions(onnxruntime_providers_webgpu PRIVATE - ORT_PLUGIN_EP_ENTRY_POINT_PREFIX=WebGpu_ - ORT_PLUGIN_EP_STATICALLY_LINKED=1) + ORT_PLUGIN_EP_STATICALLY_LINKED=1 + ORT_SKIP_API_MANUAL_INIT=1) else() # # Build WebGPU EP as a shared library @@ -94,10 +94,6 @@ add_definitions("-DONNX_NAMESPACE=onnx") add_definitions("-DONNX_USE_LITE_PROTO=1") - # This EP is its own module, so it owns the lifetime of the process-global state it initializes - # (e.g. the protobuf library) and shuts it down when the factory is released. - target_compile_definitions(onnxruntime_providers_webgpu PRIVATE ORT_PLUGIN_EP_OWNS_PROCESS_GLOBALS=1) - # Set preprocessor definitions used in onnxruntime_providers_webgpu.rc if(WIN32) set(WEBGPU_DLL_FILE_DESCRIPTION "ONNX Runtime WebGPU Provider") diff --git a/include/onnxruntime/ep/api.h b/include/onnxruntime/ep/api.h index 2520028639a4e..f7402b4b1c373 100644 --- a/include/onnxruntime/ep/api.h +++ b/include/onnxruntime/ep/api.h @@ -11,13 +11,14 @@ #include // A plugin EP built as its own shared library must not call OrtGetApiBase() itself. It receives the OrtApiBase* from -// ORT and initializes the C++ API manually, so ORT_API_MANUAL_INIT is forced on for onnxruntime_cxx_api.h. +// ORT and initializes the C++ API manually, so ORT_API_MANUAL_INIT is forced on for onnxruntime_cxx_api.h. This is +// the default because it is the safe choice for a separately built plugin. // -// When the plugin EP is statically linked into the ORT binary (ORT_PLUGIN_EP_STATICALLY_LINKED), OrtGetApiBase() is +// A build that links the plugin EP into the ORT binary defines ORT_SKIP_API_MANUAL_INIT: OrtGetApiBase() is then // available in-process and the rest of the binary is compiled without ORT_API_MANUAL_INIT. MSVC's detect_mismatch // guard in onnxruntime_cxx_api.h requires every translation unit in a binary to agree on ORT_API_MANUAL_INIT, so in // that case leave it alone and let the C++ API initialize itself. -#if !defined(ORT_PLUGIN_EP_STATICALLY_LINKED) +#if !defined(ORT_SKIP_API_MANUAL_INIT) #pragma push_macro("ORT_API_MANUAL_INIT") #undef ORT_API_MANUAL_INIT #define ORT_API_MANUAL_INIT @@ -152,9 +153,9 @@ inline void ApiInit(const OrtApiBase* ort_api_base, const char* min_ort_version throw std::runtime_error("Failed to initialize EP API: GetEpApi or GetModelEditorApi returned null."); } -#if !defined(ORT_PLUGIN_EP_STATICALLY_LINKED) - // Manual init for the C++ API. When statically linked, the C++ API initializes itself from OrtGetApiBase() with - // ORT_API_VERSION, which is the same API pointer because the EP and ORT are the same binary. +#if !defined(ORT_SKIP_API_MANUAL_INIT) + // Manual init for the C++ API. When the EP is linked into the ORT binary, the C++ API initializes itself from + // OrtGetApiBase() with ORT_API_VERSION, which is the same API pointer because the EP and ORT are the same binary. Ort::InitApi(ort_api); #endif diff --git a/onnxruntime/core/providers/webgpu/ep/api.cc b/onnxruntime/core/providers/webgpu/ep/api.cc index 344c471521321..411e823068d49 100644 --- a/onnxruntime/core/providers/webgpu/ep/api.cc +++ b/onnxruntime/core/providers/webgpu/ep/api.cc @@ -1,9 +1,10 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -// See the comment on ORT_PLUGIN_EP_STATICALLY_LINKED in include/onnxruntime/ep/api.h for why manual init of the C++ -// API is only used when this EP is built as its own shared library. -#if !defined(ORT_PLUGIN_EP_STATICALLY_LINKED) +// See the comment on ORT_SKIP_API_MANUAL_INIT in include/onnxruntime/ep/api.h for why manual init of the C++ API is +// only used when this EP is built as its own shared library. Key this off the same macro so that every translation +// unit in the binary agrees on ORT_API_MANUAL_INIT. +#if !defined(ORT_SKIP_API_MANUAL_INIT) #define ORT_API_MANUAL_INIT #include "onnxruntime_cxx_api.h" #undef ORT_API_MANUAL_INIT @@ -21,21 +22,21 @@ #include "core/session/onnxruntime_env_config_keys.h" // When this EP is statically linked into the host binary instead of being built as a separate plugin library, the -// entry points below are given a prefix (e.g. `WebGpu_`) so that multiple statically linked plugin EPs can coexist -// in one binary. The build defines ORT_PLUGIN_EP_ENTRY_POINT_PREFIX in that case. ORT core declares the prefixed -// names in onnxruntime/core/session/plugin_ep/ep_static_plugins.cc. +// entry points below are given a `WebGpu_` prefix so that multiple statically linked plugin EPs can coexist in one +// binary. ORT core declares the prefixed names in onnxruntime/core/session/plugin_ep/ep_static_plugins.cc, which +// spells them out literally, so the prefix is hardcoded on both sides rather than configured by the build. // // The shared library build must leave the entry points unprefixed, because they are resolved by exact name. -#if defined(ORT_PLUGIN_EP_ENTRY_POINT_PREFIX) +#if defined(ORT_PLUGIN_EP_STATICALLY_LINKED) #define ORT_PLUGIN_EP_ENTRY_POINT_CONCAT_IMPL(prefix, name) prefix##name #define ORT_PLUGIN_EP_ENTRY_POINT_CONCAT(prefix, name) ORT_PLUGIN_EP_ENTRY_POINT_CONCAT_IMPL(prefix, name) -#define ORT_PLUGIN_EP_ENTRY_POINT(name) ORT_PLUGIN_EP_ENTRY_POINT_CONCAT(ORT_PLUGIN_EP_ENTRY_POINT_PREFIX, name) +#define ORT_PLUGIN_EP_ENTRY_POINT(name) ORT_PLUGIN_EP_ENTRY_POINT_CONCAT(WebGpu_, name) #else #define ORT_PLUGIN_EP_ENTRY_POINT(name) name #endif // To make symbols visible on macOS/iOS. Only needed when this EP is built as a separate shared library. -#if defined(__APPLE__) && !defined(ORT_PLUGIN_EP_ENTRY_POINT_PREFIX) +#if defined(__APPLE__) && !defined(ORT_PLUGIN_EP_STATICALLY_LINKED) #define EXPORT_SYMBOL __attribute__((visibility("default"))) #else #define EXPORT_SYMBOL @@ -48,7 +49,7 @@ void CleanupKernelRegistries(); } // namespace webgpu } // namespace onnxruntime -#if defined(ORT_PLUGIN_EP_OWNS_PROCESS_GLOBALS) +#if !defined(ORT_PLUGIN_EP_STATICALLY_LINKED) namespace google { namespace protobuf { void ShutdownProtobufLibrary(); @@ -154,7 +155,7 @@ EXPORT_SYMBOL OrtStatus* ORT_PLUGIN_EP_ENTRY_POINT(ReleaseEpFactory)(OrtEpFactor // STEP.4 - Destroy the global default logger wrapper ::onnxruntime::ep::adapter::LoggingManager::DestroyDefaultLogger(); -#if defined(ORT_PLUGIN_EP_OWNS_PROCESS_GLOBALS) +#if !defined(ORT_PLUGIN_EP_STATICALLY_LINKED) // STEP.5 - Shutdown protobuf library. // Only do this when this EP owns the process-global state, i.e. when it is built as a separate library. // When statically linked into the host binary, protobuf is shared with the host, which owns its lifetime. diff --git a/onnxruntime/core/session/plugin_ep/ep_static_plugins.cc b/onnxruntime/core/session/plugin_ep/ep_static_plugins.cc index d2120b4c9a5f1..b09b6f72a40a8 100644 --- a/onnxruntime/core/session/plugin_ep/ep_static_plugins.cc +++ b/onnxruntime/core/session/plugin_ep/ep_static_plugins.cc @@ -9,9 +9,9 @@ // Entry points of the plugin execution providers that are statically linked into this build. // // A statically linked provider compiles its CreateEpFactories and ReleaseEpFactory entry points with a provider -// specific prefix (see ORT_PLUGIN_EP_ENTRY_POINT_PREFIX in the provider's api.cc) so that several providers can be -// linked into one binary without their entry points colliding. Declare them here rather than including a provider -// header, so that ORT core does not take a compile time dependency on provider internals. +// specific `WebGpu_` style prefix (see ORT_PLUGIN_EP_STATICALLY_LINKED in the provider's api.cc) so that several +// providers can be linked into one binary without their entry points colliding. Declare them here rather than +// including a provider header, so that ORT core does not take a compile time dependency on provider internals. #if defined(USE_WEBGPU) && defined(ORT_WEBGPU_STATIC_PLUGIN) extern "C" { OrtStatus* WebGpu_CreateEpFactories(const char* registration_name, const OrtApiBase* ort_api_base, From c1bc945a0e4f4d742fac245bb6d564d402d60be0 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 3 Sep 2026 12:33:33 -0700 Subject: [PATCH 045/107] Make OrtEnv::GetOrCreateInstance exception safe Static plugin EP registration was cleaned up by hand: on a failed Status the instance was unpublished, ref_count_ reset and the instance deleted. A throw from the same call skipped all of that and left a published, half-built p_instance_ for the next caller to pick up. Hold the instance in an OrtEnvPtr taken immediately after ++ref_count_ instead. That covers both exit paths and removes the manual delete and ref_count_ = 0. It is safe because OrtEnv::Release re-enters m_, which is a std::recursive_mutex already held by this thread. Also move the note about CreateAndRegisterStaticPluginEps() not taking mutex_ to the implementation; the declaration keeps the call ordering contract, which is what a caller needs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- include/onnxruntime/core/session/environment.h | 4 ---- onnxruntime/core/session/environment.cc | 4 +++- onnxruntime/core/session/ort_env.cc | 9 ++++----- 3 files changed, 7 insertions(+), 10 deletions(-) diff --git a/include/onnxruntime/core/session/environment.h b/include/onnxruntime/core/session/environment.h index 3a9f074cc895b..9f61f19e0a601 100644 --- a/include/onnxruntime/core/session/environment.h +++ b/include/onnxruntime/core/session/environment.h @@ -147,10 +147,6 @@ class Environment { * statically linked plugin EP uses the public ORT API, and any OrtEnv API it calls must be able to find the * instance. See docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/ * static_plugin_ep_registration_design.md. - * - * Note: this does not take mutex_. It is safe because the caller holds the OrtEnv creation mutex, so no other - * thread can obtain a reference to this Environment yet. Taking mutex_ here would deadlock if a plugin EP called - * an Environment API from its OrtEpFactory::GetSupportedDevices implementation. */ Status CreateAndRegisterStaticPluginEps(); diff --git a/onnxruntime/core/session/environment.cc b/onnxruntime/core/session/environment.cc index 2b045d07b4571..453b0430a1b63 100644 --- a/onnxruntime/core/session/environment.cc +++ b/onnxruntime/core/session/environment.cc @@ -645,7 +645,9 @@ Status Environment::CreateAndRegisterInternalEps() { } Status Environment::CreateAndRegisterStaticPluginEps() { - // Note: intentionally does not take mutex_. See the declaration in environment.h for why. + // Intentionally does not take mutex_. That is safe because the caller holds the OrtEnv creation mutex, so no other + // thread can obtain a reference to this Environment yet. Taking mutex_ here would deadlock if a plugin EP called an + // Environment API from its OrtEpFactory::GetSupportedDevices implementation. auto static_plugin_ep_libraries = CreateStaticPluginEpLibraries(); for (auto& ep_library : static_plugin_ep_libraries) { const std::string registration_name = ep_library->RegistrationName(); diff --git a/onnxruntime/core/session/ort_env.cc b/onnxruntime/core/session/ort_env.cc index ca855a7652f54..49ad00c3a233f 100644 --- a/onnxruntime/core/session/ort_env.cc +++ b/onnxruntime/core/session/ort_env.cc @@ -92,8 +92,10 @@ OrtEnvPtr OrtEnv::GetOrCreateInstance(const OrtEnv::LoggingManagerConstructionIn p_instance_ = new OrtEnv(std::move(env)); // Take this caller's reference before running any execution provider code below, so that if a provider - // acquires and releases an OrtEnvPtr the instance isn't destroyed out from under us. + // acquires and releases an OrtEnvPtr the instance isn't destroyed out from under us. Holding it in an OrtEnvPtr + // also unpublishes and destroys the half-built instance if anything below fails or throws. ++ref_count_; + OrtEnvPtr instance{p_instance_, OrtEnv::Release}; #if !defined(ORT_MINIMAL_BUILD) // Register statically linked plugin EPs *after* p_instance_ is published. They use the public ORT API, so any @@ -103,14 +105,11 @@ OrtEnvPtr OrtEnv::GetOrCreateInstance(const OrtEnv::LoggingManagerConstructionIn status = p_instance_->GetEnvironment().CreateAndRegisterStaticPluginEps(); if (!status.IsOK()) { - OrtEnv* failed_instance = std::exchange(p_instance_, nullptr); - ref_count_ = 0; - delete failed_instance; return OrtEnvPtr(nullptr, OrtEnv::Release); } #endif // !defined(ORT_MINIMAL_BUILD) - return OrtEnvPtr(p_instance_, OrtEnv::Release); + return instance; } ++ref_count_; From ca1dde17e3c2f5055ce14f643a25b8048e708720 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 3 Sep 2026 12:33:36 -0700 Subject: [PATCH 046/107] Select every matching device in the wasm OrtAppendExecutionProviderV2 The wrapper picked the first OrtEpDevice whose EP name matched and silently ignored the rest. An EP may report several devices that all share its name: Factory::GetSupportedDevicesImpl loops over every GPU hardware device, and the virtual device path adds its device in addition to any real ones. Collect all matches and pass the whole set, which is what SessionOptionsAppendExecutionProvider_V2 expects. Also correct the Emscripten device discovery comment. Node has never natively exposed navigator.gpu, but the userland Dawn based packages assign their GPU object onto globalThis, so the existing check does detect those. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../core/platform/emscripten/device_discovery.cc | 5 +++-- onnxruntime/wasm/api.cc | 13 ++++++++++--- onnxruntime/wasm/api.h | 5 +++-- 3 files changed, 16 insertions(+), 7 deletions(-) diff --git a/onnxruntime/core/platform/emscripten/device_discovery.cc b/onnxruntime/core/platform/emscripten/device_discovery.cc index 0b4b544a573ee..c5e4d12c52ee7 100644 --- a/onnxruntime/core/platform/emscripten/device_discovery.cc +++ b/onnxruntime/core/platform/emscripten/device_discovery.cc @@ -18,8 +18,9 @@ namespace { // also the same signal ORT Web itself uses to decide whether WebGPU is usable (see initEp() in // js/web/lib/wasm/wasm-core-impl.ts). // -// navigator is defined in both window and worker scopes, so this works on either. It is undefined, or present -// without a gpu property, in a non-browser host such as Node.js, in which case no GPU is reported. +// navigator is defined in both window and worker scopes, so this works on either. Node.js has no native WebGPU +// implementation (nodejs/node#42896 was closed as not planned), but the userland Dawn based packages install their +// GPU object onto globalThis, so this check picks those up too. Where neither is present no GPU is reported. bool IsWebGpuAvailable() { // clang-format off // The body below is JavaScript, not C++; clang-format would otherwise rewrite `!==` as `!= =`. diff --git a/onnxruntime/wasm/api.cc b/onnxruntime/wasm/api.cc index 0acbb0d8743d3..5951bd20d6b8c 100644 --- a/onnxruntime/wasm/api.cc +++ b/onnxruntime/wasm/api.cc @@ -190,15 +190,22 @@ int OrtAppendExecutionProviderV2(ort_session_options_handle_t session_options, size_t num_ep_devices = 0; RETURN_ERROR_CODE_IF_ERROR(GetEpDevices, g_env, &ep_devices, &num_ep_devices); + // An EP may report more than one device (e.g. one per GPU), and all of its devices share its name. Select all of + // them, matching the semantics of SessionOptionsAppendExecutionProvider_V2, rather than picking an arbitrary one. + std::vector selected_ep_devices; for (size_t i = 0; i < num_ep_devices; ++i) { const char* ep_name = Ort::GetApi().EpDevice_EpName(ep_devices[i]); if (ep_name != nullptr && std::strcmp(ep_name, name) == 0) { - return CHECK_STATUS(SessionOptionsAppendExecutionProvider_V2, session_options, g_env, - &ep_devices[i], static_cast(1), - provider_options_keys, provider_options_values, num_keys); + selected_ep_devices.push_back(ep_devices[i]); } } + if (!selected_ep_devices.empty()) { + return CHECK_STATUS(SessionOptionsAppendExecutionProvider_V2, session_options, g_env, + selected_ep_devices.data(), selected_ep_devices.size(), + provider_options_keys, provider_options_values, num_keys); + } + std::ostringstream message; message << "No execution provider device is registered for '" << name << "'. Registered devices:"; if (num_ep_devices == 0) { diff --git a/onnxruntime/wasm/api.h b/onnxruntime/wasm/api.h index 596ba46b73e4a..4df9fb2db95c3 100644 --- a/onnxruntime/wasm/api.h +++ b/onnxruntime/wasm/api.h @@ -98,8 +98,9 @@ int EMSCRIPTEN_KEEPALIVE OrtAppendExecutionProvider(ort_session_options_handle_t * append an execution provider for a session, selecting it as a plugin EP by name. * * Unlike OrtAppendExecutionProvider, which resolves the name against ORT's built-in EP table, this looks the - * name up among the OrtEpDevice instances registered with the environment and appends the matching one through - * SessionOptionsAppendExecutionProvider_V2. It is how a statically linked plugin EP is selected. + * name up among the OrtEpDevice instances registered with the environment and appends every device reported + * under that name through SessionOptionsAppendExecutionProvider_V2. It is how a statically linked plugin EP is + * selected. * * @param name the EP name as reported by the EP factory, e.g. "WebGpuExecutionProvider" * @returns ORT error code. If not zero, call OrtGetLastError() to get detailed error message. From 31530b5f1f62f2589daa9df7bcba153e37170d1c Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 3 Sep 2026 12:42:45 -0700 Subject: [PATCH 047/107] Record the PR review feedback round in the work log Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../static_plugin_ep_work_log.md | 60 ++++++++++++++++++- 1 file changed, 59 insertions(+), 1 deletion(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md index 1225f08399f75..dcc67f261f138 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md @@ -33,7 +33,7 @@ The implementation is complete and committed. CI coverage was added for the new | Linux/GCC build (`--use_webgpu static_plugin`) | **Green in CI** (run `33030694040`) after the `array-bounds` fix — see [GCC](#gcc-and-warnings-as-errors) | | Minimal build | Not yet exercised — see [Open items](#open-items) | | Emscripten / ORT Web | **Builds and links**, plugin registration verified in the binary — see [ORT Web](#emscripten-and-ort-web). Not yet run in a browser | -| PR | Not opened yet | +| PR | Draft [#32395](https://github.com/microsoft/onnxruntime/pull/32395); self-review feedback addressed — see [PR review feedback](#pr-review-feedback) | ## Branch @@ -637,6 +637,64 @@ no-op. The evidence is strong for the web build, where the patched `GPUQueue.sub instrument for the claimed harm. It is weaker for native, where only timing was compared and submits were never counted. +## PR review feedback + +Draft PR [#32395](https://github.com/microsoft/onnxruntime/pull/32395) collected eleven inline self-review +comments. All eleven are addressed. Two turned out to rest on a premise the code does not support, and are +recorded here because the resolution differs from the comment. + +**`OrtAppendExecutionProviderV2` should verify there is only one matching device.** It should not: a hard error +on more than one match would be wrong. `Factory::GetSupportedDevicesImpl` loops over every GPU hardware device, +and the virtual-device path adds its device *in addition to* any real ones, so several `OrtEpDevice`s +legitimately share one EP name. The real defect was the silent, arbitrary first-match pick. The fix collects +every matching device and passes the whole set, which is what `SessionOptionsAppendExecutionProvider_V2` +expects. That also settles the companion comment asking for a rename to reflect a signature mismatch — under +the new behaviour the wasm wrapper matches the underlying API, and `OrtAppendExecutionProvider` ↔ +`SessionOptionsAppendExecutionProvider` is the established naming convention in this layer, so the name stays. + +**The Emscripten device discovery should detect a GPU in Node.js some other way.** There is no better way, and +the existing check is not as blind as the comment assumed. Node has never natively exposed `navigator.gpu`; the +request was closed as not planned ([nodejs/node#42896](https://github.com/nodejs/node/issues/42896)) and there +is no `--experimental-webgpu` flag in core. WebGPU in Node comes from userland Dawn bindings, which assign onto +`globalThis`, so the existing `EM_ASM` check already picks them up. A real enumeration would need the +asynchronous `requestAdapter()`, which cannot be called from discovery. Only the comment was wrong — it claimed +Node always reports no GPU — so this is a comment-only change. The file is under `core/platform/emscripten/` +and so never affects native `onnxruntime-node`, which has its own discovery. + +Two of the remaining nine were real defects rather than cleanups: + +- `OpKernelInfo::GetAttrsImpl` for `std::vector` leaked the allocator's array, and every string + still in it, if `reserve()` or a `std::string` copy threw. It now holds the array in a `unique_ptr` whose + deleter frees each element. The deleter calls `OrtAllocator::Free` through the function pointer rather than + `Ort::AllocatorWithDefaultOptions::Free()`, which throws on failure and so must not run in a destructor. This + only matters for native plugin EPs; on web, `-sDISABLE_EXCEPTION_CATCHING` turns `bad_alloc` into an abort. +- `OrtEnv::GetOrCreateInstance` unpublished and deleted the instance by hand when static plugin EP registration + failed, but a *throw* from that registration skipped the cleanup and left a published, half-built + `p_instance_` for the next caller to pick up. The instance is now held in an `OrtEnvPtr` taken immediately + after `++ref_count_`, which removes the manual `delete` and `ref_count_ = 0` as well. This is safe because + `OrtEnv::Release` re-enters `m_`, which is a `std::recursive_mutex` already held by this thread. + +The rest were naming and comment cleanups. The most substantive: the build defined three macros that were +perfectly correlated — `ORT_PLUGIN_EP_ENTRY_POINT_PREFIX` and `ORT_PLUGIN_EP_STATICALLY_LINKED` on the static +branch, `ORT_PLUGIN_EP_OWNS_PROCESS_GLOBALS` on the shared one. The first and third are gone; everything keys +off `ORT_PLUGIN_EP_STATICALLY_LINKED`, and the `WebGpu_` prefix is hardcoded. Parameterising only the provider +side was fictional flexibility anyway, since ORT core already spells `WebGpu_CreateEpFactories` out literally +in `ep_static_plugins.cc`. + +Separately, the manual-init switch in the public header `include/onnxruntime/ep/api.h` was renamed from +`ORT_PLUGIN_EP_STATICALLY_LINKED` to `ORT_SKIP_API_MANUAL_INIT`, since it says nothing about linkage — it says +whether `OrtGetApiBase()` is reachable in-process. The negative polarity is deliberate and was kept: undefined +must continue to mean manual init *on*, which is the safe default for a separately built plugin. A positive +spelling would silently flip that for any out-of-tree plugin that relies on the header. `webgpu/ep/api.cc` has +its own copy of the `ORT_API_MANUAL_INIT` dance and is keyed off the same macro, because MSVC's +`detect_mismatch` pragma requires every translation unit in a binary to agree. + +Verification: compiled in all three affected configurations — `build\int` for the core session objects, `build\sp` +for the whole `onnxruntime_providers_webgpu` target (which force-includes the EP adapter headers, so it is what +covers `op_kernel_info.h`), and `build\wasm_sp` for `wasm/api.cc` and the Emscripten `device_discovery.cc`. On the +RTX 5060 Ti machine, `onnxruntime_test_all --gtest_filter=InferenceSessionTests.WebGpu*` passes all three, and +`onnxruntime_provider_test` passes with no failures. + ## Open items 1. ~~**Fix the Linux leg.**~~ **Done.** The batch-3 `array-bounds` break is fixed at the source, and a local From 53bf4985420671d3c05f33cd456f8ebd6ef17f63 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 3 Sep 2026 13:21:30 -0700 Subject: [PATCH 048/107] Pass EP devices explicitly to the wasm OrtAppendExecutionProviderV2 The wasm wrapper took an EP name and resolved it to a single OrtEpDevice by picking the first name match, which is silently arbitrary: a WebGPU factory can report several devices under one EP name (one per GPU, plus the virtual device in addition to any real ones). Make the export mirror SessionOptionsAppendExecutionProvider_V2 instead - it now takes an OrtEpDevice array and a count, and is a pass-through. Selection policy moves to JavaScript, supported by two new thin exports, OrtGetEpDevices and OrtEpDevice_EpName. session-options.ts collects the devices into a map keyed by EP name, passes every match, and can name the registered EPs when none match. Filtering in C++ was rejected: GetEpDevices returns an environment-owned array, and a filtered result would have to be materialised, inventing an ownership contract this layer does not otherwise have. Also add the three _Ort* exports to the no-underscore-dangle allow-list in js/eslint.config.mjs; _OrtAppendExecutionProviderV2 was missing from it already and was failing npm run lint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../static_plugin_ep_work_log.md | 35 +++++++-- js/eslint.config.mjs | 3 + js/web/lib/wasm/session-options.ts | 78 ++++++++++++++++--- js/web/lib/wasm/wasm-types.ts | 5 +- onnxruntime/wasm/api.cc | 42 +++------- onnxruntime/wasm/api.h | 38 +++++++-- 6 files changed, 149 insertions(+), 52 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md index dcc67f261f138..1106bbe59e451 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md @@ -646,11 +646,36 @@ recorded here because the resolution differs from the comment. **`OrtAppendExecutionProviderV2` should verify there is only one matching device.** It should not: a hard error on more than one match would be wrong. `Factory::GetSupportedDevicesImpl` loops over every GPU hardware device, and the virtual-device path adds its device *in addition to* any real ones, so several `OrtEpDevice`s -legitimately share one EP name. The real defect was the silent, arbitrary first-match pick. The fix collects -every matching device and passes the whole set, which is what `SessionOptionsAppendExecutionProvider_V2` -expects. That also settles the companion comment asking for a rename to reflect a signature mismatch — under -the new behaviour the wasm wrapper matches the underlying API, and `OrtAppendExecutionProvider` ↔ -`SessionOptionsAppendExecutionProvider` is the established naming convention in this layer, so the name stays. +legitimately share one EP name. The real defect was the silent, arbitrary first-match pick. + +The fix is bigger than the comment asked for, because a companion comment objected that the wrapper's name +claims a kinship with `SessionOptionsAppendExecutionProvider_V2` that its signature did not honour. Both are +settled by moving EP-device selection out of C++ and into JavaScript, so that the wasm export really is the +underlying API: + +- `OrtAppendExecutionProviderV2` now takes an `OrtEpDevice*` array and a count, exactly like the C API, and is + a straight pass-through. The name is kept, because now the signature does match. +- Two new exports supply what JavaScript needs to build that array: `OrtGetEpDevices`, a thin wrapper over + `GetEpDevices`, and `OrtEpDevice_EpName`, a 1:1 mirror of `EpDevice_EpName`. +- `session-options.ts` gains `getEpDevicesByEpName()`, which walks the returned array once into a + `Map`. Selection policy — currently "every device whose EP name matches" — lives there, and + the not-found error can list the registered names because the map has them. + +Filtering on the C++ side was considered and rejected. `GetEpDevices` returns an array *owned by the +environment*; passing the pointer straight through keeps it borrowed. Any C++-side filter would have to +materialise a new array and invent an ownership contract that this layer does not otherwise have. The selected +subset has to be marshalled into wasm memory by the caller either way, and `setExecutionProviders` already has +an `allocs` list that does exactly this for the EP option keys and values. + +Two traps in the JavaScript half, neither of which the type checker catches: + +- Pointers read out of wasm memory with `getValue(ptr, '*')` must be wrapped in `Number(...)`. Under `ORT_WASM64` + that call reads `HEAP64` and yields a **BigInt**, and `BigInt + Number` pointer arithmetic throws a + `TypeError`. Values *returned from* an exported function need no wrapping — Emscripten converts those at the + call boundary. `jsep/init.ts` and `wasm-core-impl.ts` follow the same split. +- Every new `_Ort*` export must be added to the `no-underscore-dangle` allow-list in `js/eslint.config.mjs`, or + `npm run lint` fails. The earlier commit that introduced `_OrtAppendExecutionProviderV2` missed this and left + the branch red; that is fixed here too. **The Emscripten device discovery should detect a GPU in Node.js some other way.** There is no better way, and the existing check is not as blind as the comment assumed. Node has never natively exposed `navigator.gpu`; the diff --git a/js/eslint.config.mjs b/js/eslint.config.mjs index 9a5edd2e6761e..9ae9f36362dba 100644 --- a/js/eslint.config.mjs +++ b/js/eslint.config.mjs @@ -273,6 +273,7 @@ export default [ '_OrtAddRunConfigEntry', '_OrtAddSessionConfigEntry', '_OrtAppendExecutionProvider', + '_OrtAppendExecutionProviderV2', '_OrtBindInput', '_OrtBindOutput', '_OrtClearBoundOutputs', @@ -282,7 +283,9 @@ export default [ '_OrtCreateSessionOptions', '_OrtCreateTensor', '_OrtEndProfiling', + '_OrtEpDevice_EpName', '_OrtFree', + '_OrtGetEpDevices', '_OrtGetInputName', '_OrtGetInputOutputCount', '_OrtGetInputOutputMetadata', diff --git a/js/web/lib/wasm/session-options.ts b/js/web/lib/wasm/session-options.ts index 827d87c6bbf3b..c53b77c01d028 100644 --- a/js/web/lib/wasm/session-options.ts +++ b/js/web/lib/wasm/session-options.ts @@ -70,6 +70,44 @@ const appendEpOption = (epOptions: Array<[number, number]>, key: string, value: epOptions.push([keyDataOffset, valueDataOffset]); }; +/** + * Get the OrtEpDevice instances registered with the environment, grouped by execution provider name. + * + * An execution provider may register more than one device, e.g. one per GPU, and all of them share its name. + * + * @returns a map of execution provider name to the OrtEpDevice pointers registered under it. + */ +const getEpDevicesByEpName = (): Map => { + const wasm = getInstance(); + const stack = wasm.stackSave(); + try { + const epDevicesPtr = wasm.stackAlloc(wasm.PTR_SIZE); + const numEpDevicesPtr = wasm.stackAlloc(wasm.PTR_SIZE); + if (wasm._OrtGetEpDevices(epDevicesPtr, numEpDevicesPtr) !== 0) { + checkLastError("Can't get execution provider devices."); + } + // The array is owned by the environment, so only the pointers are read out here. + // getValue() with '*' returns a BigInt in a wasm64 build, so convert before doing pointer arithmetic. + const epDevices = Number(wasm.getValue(epDevicesPtr, '*')); + const numEpDevices = Number(wasm.getValue(numEpDevicesPtr, '*')); + + const epDevicesByEpName = new Map(); + for (let i = 0; i < numEpDevices; i++) { + const epDevice = Number(wasm.getValue(epDevices + i * wasm.PTR_SIZE, '*')); + const epName = wasm.UTF8ToString(wasm._OrtEpDevice_EpName(epDevice)); + const devices = epDevicesByEpName.get(epName); + if (devices) { + devices.push(epDevice); + } else { + epDevicesByEpName.set(epName, [epDevice]); + } + } + return epDevicesByEpName; + } finally { + wasm.stackRestore(stack); + } +}; + const setExecutionProviders = async ( sessionOptionsHandle: number, sessionOptions: InferenceSession.SessionOptions, @@ -190,7 +228,6 @@ const setExecutionProviders = async ( throw new Error(`not supported execution provider: ${epName}`); } - const epNameDataOffset = allocWasmString(epName, allocs); const epOptionsCount = epOptions.length; let keysOffset = 0; let valuesOffset = 0; @@ -204,19 +241,42 @@ const setExecutionProviders = async ( getInstance().setValue(valuesOffset + i * getInstance().PTR_SIZE, epOptions[i][1], '*'); } } - const appendFn = selectAsPluginEp - ? getInstance()._OrtAppendExecutionProviderV2 - : getInstance()._OrtAppendExecutionProvider; - if ( - (await appendFn.call( - getInstance(), + + let appendErrorCode: number; + if (selectAsPluginEp) { + const epDevicesByEpName = getEpDevicesByEpName(); + const epDevices = epDevicesByEpName.get(epName); + if (!epDevices) { + const registered = [...epDevicesByEpName.keys()].join(', ') || '(none)'; + throw new Error(`no execution provider device is registered for: ${epName}. registered: ${registered}.`); + } + + const epDevicesOffset = getInstance()._malloc(epDevices.length * getInstance().PTR_SIZE); + allocs.push(epDevicesOffset); + for (let i = 0; i < epDevices.length; i++) { + getInstance().setValue(epDevicesOffset + i * getInstance().PTR_SIZE, epDevices[i], '*'); + } + + appendErrorCode = await getInstance()._OrtAppendExecutionProviderV2( + sessionOptionsHandle, + epDevicesOffset, + epDevices.length, + keysOffset, + valuesOffset, + epOptionsCount, + ); + } else { + const epNameDataOffset = allocWasmString(epName, allocs); + appendErrorCode = await getInstance()._OrtAppendExecutionProvider( sessionOptionsHandle, epNameDataOffset, keysOffset, valuesOffset, epOptionsCount, - )) !== 0 - ) { + ); + } + + if (appendErrorCode !== 0) { checkLastError(`Can't append execution provider: ${epName}.`); } } diff --git a/js/web/lib/wasm/wasm-types.ts b/js/web/lib/wasm/wasm-types.ts index 6e6d1676d503e..02e1a0051c6ff 100644 --- a/js/web/lib/wasm/wasm-types.ts +++ b/js/web/lib/wasm/wasm-types.ts @@ -415,11 +415,14 @@ export interface OrtInferenceAPIs { ): Promise; _OrtAppendExecutionProviderV2( sessionOptionsHandle: number, - name: number, + epDevices: number, + numEpDevices: number, providerOptionsKeys: number, providerOptionsValues: number, numKeys: number, ): Promise; + _OrtGetEpDevices(epDevices: number, numEpDevices: number): number; + _OrtEpDevice_EpName(epDevice: number): number; _OrtAddFreeDimensionOverride(sessionOptionsHandle: number, name: number, dim: number): number; _OrtAddSessionConfigEntry(sessionOptionsHandle: number, configKey: number, configValue: number): number; _OrtReleaseSessionOptions(sessionOptionsHandle: number): number; diff --git a/onnxruntime/wasm/api.cc b/onnxruntime/wasm/api.cc index 5951bd20d6b8c..67d13c884c62e 100644 --- a/onnxruntime/wasm/api.cc +++ b/onnxruntime/wasm/api.cc @@ -181,40 +181,22 @@ int OrtAppendExecutionProvider(ort_session_options_handle_t session_options, return CHECK_STATUS(SessionOptionsAppendExecutionProvider, session_options, name, provider_options_keys, provider_options_values, num_keys); } +int OrtGetEpDevices(const ort_ep_device_handle_t** ep_devices, size_t* num_ep_devices) { + return CHECK_STATUS(GetEpDevices, g_env, ep_devices, num_ep_devices); +} + +const char* OrtEpDevice_EpName(ort_ep_device_handle_t ep_device) { + return Ort::GetApi().EpDevice_EpName(ep_device); +} + int OrtAppendExecutionProviderV2(ort_session_options_handle_t session_options, - const char* name, + ort_ep_device_handle_t* ep_devices, + size_t num_ep_devices, const char* const* provider_options_keys, const char* const* provider_options_values, size_t num_keys) { - const OrtEpDevice* const* ep_devices = nullptr; - size_t num_ep_devices = 0; - RETURN_ERROR_CODE_IF_ERROR(GetEpDevices, g_env, &ep_devices, &num_ep_devices); - - // An EP may report more than one device (e.g. one per GPU), and all of its devices share its name. Select all of - // them, matching the semantics of SessionOptionsAppendExecutionProvider_V2, rather than picking an arbitrary one. - std::vector selected_ep_devices; - for (size_t i = 0; i < num_ep_devices; ++i) { - const char* ep_name = Ort::GetApi().EpDevice_EpName(ep_devices[i]); - if (ep_name != nullptr && std::strcmp(ep_name, name) == 0) { - selected_ep_devices.push_back(ep_devices[i]); - } - } - - if (!selected_ep_devices.empty()) { - return CHECK_STATUS(SessionOptionsAppendExecutionProvider_V2, session_options, g_env, - selected_ep_devices.data(), selected_ep_devices.size(), - provider_options_keys, provider_options_values, num_keys); - } - - std::ostringstream message; - message << "No execution provider device is registered for '" << name << "'. Registered devices:"; - if (num_ep_devices == 0) { - message << " (none)"; - } - for (size_t i = 0; i < num_ep_devices; ++i) { - message << " '" << Ort::GetApi().EpDevice_EpName(ep_devices[i]) << "'"; - } - return CheckStatus(Ort::GetApi().CreateStatus(ORT_NOT_FOUND, message.str().c_str())); + return CHECK_STATUS(SessionOptionsAppendExecutionProvider_V2, session_options, g_env, + ep_devices, num_ep_devices, provider_options_keys, provider_options_values, num_keys); } int OrtAddFreeDimensionOverride(ort_session_options_handle_t session_options, diff --git a/onnxruntime/wasm/api.h b/onnxruntime/wasm/api.h index 4df9fb2db95c3..b18fcf2e337b6 100644 --- a/onnxruntime/wasm/api.h +++ b/onnxruntime/wasm/api.h @@ -31,6 +31,9 @@ using ort_run_options_handle_t = OrtRunOptions*; struct OrtValue; using ort_tensor_handle_t = OrtValue*; +struct OrtEpDevice; +using ort_ep_device_handle_t = const OrtEpDevice*; + #ifdef ENABLE_TRAINING_APIS struct OrtTrainingSession; using ort_training_session_handle_t = OrtTrainingSession*; @@ -95,18 +98,39 @@ int EMSCRIPTEN_KEEPALIVE OrtAppendExecutionProvider(ort_session_options_handle_t size_t num_keys); /** - * append an execution provider for a session, selecting it as a plugin EP by name. + * get the execution provider devices registered with the environment. + * + * The array is owned by the environment and remains valid for its lifetime. The caller must not free it. + * + * @param ep_devices receives a pointer to the array of ort_ep_device_handle_t + * @param num_ep_devices receives the number of entries in the array + * @returns ORT error code. If not zero, call OrtGetLastError() to get detailed error message. + */ +int EMSCRIPTEN_KEEPALIVE OrtGetEpDevices(const ort_ep_device_handle_t** ep_devices, size_t* num_ep_devices); + +/** + * get the execution provider name that an execution provider device is registered under. + * + * More than one device may share an EP name, e.g. when an EP reports one device per GPU. + * + * @returns the name, owned by the environment. It remains valid for the lifetime of the environment. + */ +const char* EMSCRIPTEN_KEEPALIVE OrtEpDevice_EpName(ort_ep_device_handle_t ep_device); + +/** + * append an execution provider for a session, selecting it by execution provider device. * - * Unlike OrtAppendExecutionProvider, which resolves the name against ORT's built-in EP table, this looks the - * name up among the OrtEpDevice instances registered with the environment and appends every device reported - * under that name through SessionOptionsAppendExecutionProvider_V2. It is how a statically linked plugin EP is - * selected. + * Unlike OrtAppendExecutionProvider, which resolves an EP name against ORT's built-in EP table, this selects + * from the devices registered with the environment, which is how a plugin EP is selected. Use OrtGetEpDevices + * and OrtEpDevice_EpName to choose the devices to pass in. * - * @param name the EP name as reported by the EP factory, e.g. "WebGpuExecutionProvider" + * @param ep_devices the devices to select, all of which must belong to the same execution provider + * @param num_ep_devices the number of entries in ep_devices * @returns ORT error code. If not zero, call OrtGetLastError() to get detailed error message. */ int EMSCRIPTEN_KEEPALIVE OrtAppendExecutionProviderV2(ort_session_options_handle_t session_options, - const char* name, + ort_ep_device_handle_t* ep_devices, + size_t num_ep_devices, const char* const* provider_options_keys, const char* const* provider_options_values, size_t num_keys); From 69d8fc3aea6dcc35376565908c589c2e60bdf2e4 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 3 Sep 2026 14:54:46 -0700 Subject: [PATCH 049/107] Record end-to-end verification of the EP-device-array path Relinked build\web_plugin and ran the browser harness. Outputs are bit-identical to the pre-rework plugin build and GPU submits per run are unchanged at 20; a patched bogus EP name now fails in JavaScript and names the registered EPs, which is what proves the device enumeration actually ran. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../static_plugin_ep_work_log.md | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md index 1106bbe59e451..9db19e34dddee 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md @@ -677,6 +677,29 @@ Two traps in the JavaScript half, neither of which the type checker catches: `npm run lint` fails. The earlier commit that introduced `_OrtAppendExecutionProviderV2` missed this and left the branch red; that is fixed here too. +The device-array path was then run, not just compiled. `build\web_plugin` was relinked and a fresh `plugin` side +built, and two facts were checked against the pre-rework `plugin` results: + +| endpoint | before | after | +| --- | --- | --- | +| `bench_dispatch` output sample | `-0.547067046165466, 0.458441495895386, 0.369816303253174, 0.281192660331726` | identical | +| GPU submits per run | 20 | 20 | + +Submit counts are deterministic, so an identical count plus bit-identical outputs is a much stronger check than +any timing. The negative control is the load-bearing one: patching the bundle's EP name to +`BogusExecutionProviderX` now fails in JavaScript with + +``` +no execution provider device is registered for: BogusExecutionProviderX. +registered: CPUExecutionProvider, WebGpuExecutionProvider. +``` + +That message can only be produced by walking a real device array, so it proves `OrtGetEpDevices`, +the `Number(getValue(...))` pointer arithmetic, `OrtEpDevice_EpName` and the grouping all work — the parts a +successful run would otherwise exercise silently. Confirming that the two new exports are also *not* +asyncify-wrapped: `_OrtAppendExecutionProviderV2` appears twice in the generated `.mjs` (export plus JSPI +wrapper) while `_OrtGetEpDevices` and `_OrtEpDevice_EpName` appear once each. + **The Emscripten device discovery should detect a GPU in Node.js some other way.** There is no better way, and the existing check is not as blind as the comment assumed. Node has never natively exposed `navigator.gpu`; the request was closed as not planned ([nodejs/node#42896](https://github.com/nodejs/node/issues/42896)) and there From 765c59a087930f43c477b6c607380a0d0b39c791 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 3 Sep 2026 16:11:25 -0700 Subject: [PATCH 050/107] Select a single EP device when appending the WebGPU plugin EP The WebGPU EP factory backs a session with one Dawn device and rejects any device count other than 1, so passing every name match would fail as soon as a second OrtEpDevice appeared. Pass the first match instead, with a note on why one device is required and why taking the first is safe. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../static_plugin_ep_work_log.md | 28 ++++++++++++++++--- js/web/lib/wasm/session-options.ts | 20 +++++++++---- 2 files changed, 39 insertions(+), 9 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md index 9db19e34dddee..787bd5659956e 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md @@ -643,10 +643,12 @@ Draft PR [#32395](https://github.com/microsoft/onnxruntime/pull/32395) collected comments. All eleven are addressed. Two turned out to rest on a premise the code does not support, and are recorded here because the resolution differs from the comment. -**`OrtAppendExecutionProviderV2` should verify there is only one matching device.** It should not: a hard error +**`OrtAppendExecutionProviderV2` should verify there is only one matching device.** It should not: erroring out on more than one match would be wrong. `Factory::GetSupportedDevicesImpl` loops over every GPU hardware device, and the virtual-device path adds its device *in addition to* any real ones, so several `OrtEpDevice`s -legitimately share one EP name. The real defect was the silent, arbitrary first-match pick. +legitimately share one EP name. The comment was right that a single device has to be chosen, but the choice +belongs to the caller, not to the wasm wrapper — and the real defect was that the pick was silent and +undocumented rather than that it happened at all. The fix is bigger than the comment asked for, because a companion comment objected that the wrapper's name claims a kinship with `SessionOptionsAppendExecutionProvider_V2` that its signature did not honour. Both are @@ -658,8 +660,26 @@ underlying API: - Two new exports supply what JavaScript needs to build that array: `OrtGetEpDevices`, a thin wrapper over `GetEpDevices`, and `OrtEpDevice_EpName`, a 1:1 mirror of `EpDevice_EpName`. - `session-options.ts` gains `getEpDevicesByEpName()`, which walks the returned array once into a - `Map`. Selection policy — currently "every device whose EP name matches" — lives there, and - the not-found error can list the registered names because the map has them. + `Map`. Selection policy lives there, and the not-found error can list the registered names + because the map has them. + +The selection policy is to pass the **first** matching device. That is necessary because the WebGPU EP backs a +session with a single Dawn device, so `Factory::CreateEpImpl` rejects any device count other than 1 — passing +every match would break the moment a second device appeared. It is safe because the factory reads nothing from +the device except whether it is virtual: which GPU gets used is decided by the adapter ORT Web hands to Dawn, +not by the `OrtEpDevice`. Emscripten device discovery reports a single GPU entry derived from `navigator.gpu` +(there is no per-GPU enumeration in a browser, and `requestAdapter` takes only `powerPreference` / +`forceFallbackAdapter` hints), so in practice there is exactly one real candidate. A second entry appears only +when virtual devices are enabled, and the factory registers those after the real ones, which `Environment` +preserves. Even if that order changed, EP creation would fail with the explicit "selected on a virtual GPU +device" error rather than silently running on the wrong device. + +Selecting an `OrtEpDevice` from JavaScript was considered and is not worth building. It would need a public +enumeration API in `onnxruntime-common` plus a set of new wasm accessors for the device descriptor, and it +would still not choose a GPU, because the browser does not expose one to choose. Applications that need to +control the device already can: `env.webgpu.adapter` and the WebGPU EP's `device` option accept a +caller-created `GPUAdapter`/`GPUDevice`, which is both the browser-sanctioned mechanism and more expressive +than picking an `OrtEpDevice`. Filtering on the C++ side was considered and rejected. `GetEpDevices` returns an array *owned by the environment*; passing the pointer straight through keeps it borrowed. Any C++-side filter would have to diff --git a/js/web/lib/wasm/session-options.ts b/js/web/lib/wasm/session-options.ts index c53b77c01d028..85db499585efa 100644 --- a/js/web/lib/wasm/session-options.ts +++ b/js/web/lib/wasm/session-options.ts @@ -251,16 +251,26 @@ const setExecutionProviders = async ( throw new Error(`no execution provider device is registered for: ${epName}. registered: ${registered}.`); } - const epDevicesOffset = getInstance()._malloc(epDevices.length * getInstance().PTR_SIZE); + // Select a single OrtEpDevice, even though more than one may match the EP name. + // + // Necessary: the WebGPU EP backs a session with one Dawn device, so its factory rejects any device + // count other than 1 (see Factory::CreateEpImpl in core/providers/webgpu/ep/factory.cc). + // + // Safe to take the first: the factory reads nothing from the device except whether it is virtual, and + // Emscripten device discovery reports a single GPU entry derived from navigator.gpu, so there is at + // most one real candidate. Which GPU is used is decided by the adapter ORT Web hands to Dawn, not by + // this device (see env.webgpu.adapter and the WebGPU EP `device` option). A second entry only appears + // when virtual devices are enabled, and the factory registers those after the real ones, so the first + // is still a real device. Should that order ever change, EP creation fails with an explicit + // "selected on a virtual GPU device" error rather than silently running on the wrong device. + const epDevicesOffset = getInstance()._malloc(getInstance().PTR_SIZE); allocs.push(epDevicesOffset); - for (let i = 0; i < epDevices.length; i++) { - getInstance().setValue(epDevicesOffset + i * getInstance().PTR_SIZE, epDevices[i], '*'); - } + getInstance().setValue(epDevicesOffset, epDevices[0], '*'); appendErrorCode = await getInstance()._OrtAppendExecutionProviderV2( sessionOptionsHandle, epDevicesOffset, - epDevices.length, + 1, keysOffset, valuesOffset, epOptionsCount, From c6ba17d7ae60de179a2c1a684d0e0ad6bc064127 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 3 Sep 2026 16:13:39 -0700 Subject: [PATCH 051/107] Record re-verification of the single-device EP selection Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../static_plugin_ep_work_log.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md index 787bd5659956e..f873961dd6008 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md @@ -720,6 +720,13 @@ successful run would otherwise exercise silently. Confirming that the two new ex asyncify-wrapped: `_OrtAppendExecutionProviderV2` appears twice in the generated `.mjs` (export plus JSPI wrapper) while `_OrtGetEpDevices` and `_OrtEpDevice_EpName` appear once each. +Narrowing the selection to a single device was re-verified the same way, against the same endpoint: the output +sample stayed bit-identical and `submitsPerRun` stayed a constant 20, with `300 + 61 x 20 = 1520` total submits +over the 10 warmup, 1 first and 50 steady runs. This machine registers one WebGPU device, so the change is +behaviour-preserving here by construction; what the run establishes is that the rewritten append block still +selects the WebGPU EP and produces identical results, not that the multi-device case is exercised. That case +needs virtual devices enabled, which ORT Web has no way to turn on today. + **The Emscripten device discovery should detect a GPU in Node.js some other way.** There is no better way, and the existing check is not as blind as the comment assumed. Node has never natively exposed `navigator.gpu`; the request was closed as not planned ([nodejs/node#42896](https://github.com/nodejs/node/issues/42896)) and there From 6fd4ff1b0042859aa1f3c2dbd33cd3bc98d972b8 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Fri, 4 Sep 2026 14:17:25 -0700 Subject: [PATCH 052/107] Rename api.h skip init macro to ORT_PLUGIN_EP_SKIP_API_MANUAL_INIT, update comments. --- cmake/onnxruntime_providers_webgpu.cmake | 6 +++--- .../static_plugin_ep_work_log.md | 2 +- include/onnxruntime/ep/api.h | 11 +++++++---- onnxruntime/core/providers/webgpu/ep/api.cc | 13 ++----------- 4 files changed, 13 insertions(+), 19 deletions(-) diff --git a/cmake/onnxruntime_providers_webgpu.cmake b/cmake/onnxruntime_providers_webgpu.cmake index 796f1d28a19ce..ab29c7a01f88c 100644 --- a/cmake/onnxruntime_providers_webgpu.cmake +++ b/cmake/onnxruntime_providers_webgpu.cmake @@ -51,11 +51,11 @@ # Prefix the plugin EP entry points so that multiple statically linked plugin EPs can coexist in one binary. # ORT core declares the prefixed names in onnxruntime/core/session/plugin_ep/ep_static_plugins.cc. # - # ORT_SKIP_API_MANUAL_INIT tells the shared plugin EP headers that OrtGetApiBase() is available in-process, - # so the C++ API must not be built with ORT_API_MANUAL_INIT (the rest of the binary is not). + # ORT_PLUGIN_EP_SKIP_API_MANUAL_INIT tells the shared plugin EP headers that OrtGetApiBase() is available + # in-process, so the C++ API must not be built with ORT_API_MANUAL_INIT (the rest of the binary is not). target_compile_definitions(onnxruntime_providers_webgpu PRIVATE ORT_PLUGIN_EP_STATICALLY_LINKED=1 - ORT_SKIP_API_MANUAL_INIT=1) + ORT_PLUGIN_EP_SKIP_API_MANUAL_INIT=1) else() # # Build WebGPU EP as a shared library diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md index f873961dd6008..2bbd46a1a3b97 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md @@ -757,7 +757,7 @@ side was fictional flexibility anyway, since ORT core already spells `WebGpu_Cre in `ep_static_plugins.cc`. Separately, the manual-init switch in the public header `include/onnxruntime/ep/api.h` was renamed from -`ORT_PLUGIN_EP_STATICALLY_LINKED` to `ORT_SKIP_API_MANUAL_INIT`, since it says nothing about linkage — it says +`ORT_PLUGIN_EP_STATICALLY_LINKED` to `ORT_PLUGIN_EP_SKIP_API_MANUAL_INIT`, since it says nothing about linkage — it says whether `OrtGetApiBase()` is reachable in-process. The negative polarity is deliberate and was kept: undefined must continue to mean manual init *on*, which is the safe default for a separately built plugin. A positive spelling would silently flip that for any out-of-tree plugin that relies on the header. `webgpu/ep/api.cc` has diff --git a/include/onnxruntime/ep/api.h b/include/onnxruntime/ep/api.h index f7402b4b1c373..93425ca5779e6 100644 --- a/include/onnxruntime/ep/api.h +++ b/include/onnxruntime/ep/api.h @@ -10,15 +10,18 @@ #include #include +// Users of this header can explicitly define ORT_PLUGIN_EP_SKIP_API_MANUAL_INIT to skip manual ORT C++ API +// initialization. +// // A plugin EP built as its own shared library must not call OrtGetApiBase() itself. It receives the OrtApiBase* from // ORT and initializes the C++ API manually, so ORT_API_MANUAL_INIT is forced on for onnxruntime_cxx_api.h. This is // the default because it is the safe choice for a separately built plugin. // -// A build that links the plugin EP into the ORT binary defines ORT_SKIP_API_MANUAL_INIT: OrtGetApiBase() is then -// available in-process and the rest of the binary is compiled without ORT_API_MANUAL_INIT. MSVC's detect_mismatch +// A build that links the plugin EP into the ORT binary defines ORT_PLUGIN_EP_SKIP_API_MANUAL_INIT: OrtGetApiBase() is +// then available in-process and the rest of the binary is compiled without ORT_API_MANUAL_INIT. MSVC's detect_mismatch // guard in onnxruntime_cxx_api.h requires every translation unit in a binary to agree on ORT_API_MANUAL_INIT, so in // that case leave it alone and let the C++ API initialize itself. -#if !defined(ORT_SKIP_API_MANUAL_INIT) +#if !defined(ORT_PLUGIN_EP_SKIP_API_MANUAL_INIT) #pragma push_macro("ORT_API_MANUAL_INIT") #undef ORT_API_MANUAL_INIT #define ORT_API_MANUAL_INIT @@ -153,7 +156,7 @@ inline void ApiInit(const OrtApiBase* ort_api_base, const char* min_ort_version throw std::runtime_error("Failed to initialize EP API: GetEpApi or GetModelEditorApi returned null."); } -#if !defined(ORT_SKIP_API_MANUAL_INIT) +#if !defined(ORT_PLUGIN_EP_SKIP_API_MANUAL_INIT) // Manual init for the C++ API. When the EP is linked into the ORT binary, the C++ API initializes itself from // OrtGetApiBase() with ORT_API_VERSION, which is the same API pointer because the EP and ORT are the same binary. Ort::InitApi(ort_api); diff --git a/onnxruntime/core/providers/webgpu/ep/api.cc b/onnxruntime/core/providers/webgpu/ep/api.cc index 411e823068d49..5447f19aa0368 100644 --- a/onnxruntime/core/providers/webgpu/ep/api.cc +++ b/onnxruntime/core/providers/webgpu/ep/api.cc @@ -1,22 +1,13 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -// See the comment on ORT_SKIP_API_MANUAL_INIT in include/onnxruntime/ep/api.h for why manual init of the C++ API is -// only used when this EP is built as its own shared library. Key this off the same macro so that every translation -// unit in the binary agrees on ORT_API_MANUAL_INIT. -#if !defined(ORT_SKIP_API_MANUAL_INIT) -#define ORT_API_MANUAL_INIT -#include "onnxruntime_cxx_api.h" -#undef ORT_API_MANUAL_INIT -#else -#include "onnxruntime_cxx_api.h" -#endif - #include #include #include #include +#include "ep/api.h" + #include "core/platform/env_var.h" #include "core/providers/webgpu/ep/factory.h" #include "core/session/onnxruntime_env_config_keys.h" From c2a6d47c1d3e9b8634ddd30cac94dd5f5e987073 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Fri, 4 Sep 2026 15:00:04 -0700 Subject: [PATCH 053/107] docs: cover native runtime performance and conformance in success criteria The performance criterion applied only to onnxruntime-web, but the per-node cost of routing kernel work through the C API adapters is a property of the adapter boundary rather than of linkage mode, so native plugin consumers pay it too. Generalize the criterion to all supported consumers, keeping the WebAssembly size budget as a web-specific addition, and name both comparison endpoints: the plugin path against the current built-in implementation. Make conformance explicit alongside it. Extend the plugin-boundary baselines work package to measure native inference latency as well, and widen its latency criterion to cover the native shared library in addition to static registration. Also correct that work package's timing. It said baselines had to be captured before any plugin-path change landed, but the shared library plugin form already ships. The real constraint is that the non-plugin path is the comparison, so the numbers must be taken before package 7 removes it, which the existing convergence rule already ensures. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- ...lugin_boundary_and_web_integration_workstream.md | 13 ++++++++----- .../webgpu_ep_extraction/webgpu_ep_extraction.md | 8 +++++--- 2 files changed, 13 insertions(+), 8 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md index 69b9b1c94235c..8a975f8edb85c 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md @@ -136,8 +136,9 @@ The bridge should not expose unrelated ORT private implementation details. ## Work packages -1. **Size and latency baselines:** measure `onnxruntime-web` WebAssembly size and inference latency before any - plugin-path change lands, since the size and latency completion criteria are relative to them. +1. **Size and latency baselines:** measure `onnxruntime-web` WebAssembly size and inference latency, and native + inference latency, on the non-plugin path while it still exists, since the size and latency completion criteria + compare the plugin path against those numbers. 2. **Static registration core:** implement and contract-test generic static factory registration. 3. **Global lifetime contract:** inventory process-global state and implement safe ownership and teardown rules. 4. **Emscripten prototype:** compile the plugin path statically and run a small model. @@ -145,8 +146,9 @@ The bridge should not expose unrelated ORT private implementation details. 6. **Browser bridge:** specify and prototype object and lifetime exchange. 7. **Parity and retirement:** run the existing suite through the plugin path and remove the legacy path. -Baselines come first, because they cannot be captured once the plugin path has changed anything. Packages 2 through 6 -can then proceed largely in parallel. Legacy-path removal waits for their convergence. +Packages 1 through 6 can proceed largely in parallel, and legacy-path removal waits for their convergence. That +ordering matters for the baselines in particular: the non-plugin path is the comparison, so once package 7 removes it +the numbers can no longer be captured. ## Interfaces with other workstreams @@ -174,7 +176,8 @@ can then proceed largely in parallel. Legacy-path removal waits for their conver - Browser object ownership and lifecycle are documented and tested. - Reduced WebAssembly builds retain required plugin infrastructure within accepted size budgets, measured against the baselines established before the work begins. -- Inference latency through static plugin registration stays within an accepted tolerance of the same baselines. +- Inference latency stays within an accepted tolerance of the same baselines, for both static plugin registration and + the native shared-library plugin. - The direct `IExecutionProvider` WebGPU path is removed. ## Open questions diff --git a/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md b/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md index c60d46d968983..06b1f8e3a9868 100644 --- a/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md +++ b/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md @@ -127,13 +127,15 @@ The other convergence points between workstreams: These are the outcomes that show the whole effort is complete: -- Static and shared builds execute the same provider implementation, through the public plugin EP API, against the - same operator tests. +- Static and shared builds execute the same provider implementation through the public plugin EP API, and pass the + same conformance suite. - The ORT repository retains provider integration shims only, and no longer contains WebGPU EP implementation, build, or packaging inputs. - The external repository owns WebGPU code, dependencies, tests, packages, and releases. - ORT updates its pinned WebGPU revision through a routine dependency update. -- `onnxruntime-web` preserves supported functionality and accepted size and performance characteristics. +- Supported consumers keep their functionality, and inference performance on the plugin path stays within an accepted + tolerance of the current built-in implementation. `onnxruntime-web` additionally stays within its WebAssembly size + budget. - Native ORT packages remain usable without installing WebGPU. - Existing Node WebGPU users have a documented and tested migration path. - Compatibility failures produce clear build-time or registration-time diagnostics. From 4aef27ccb895f98eea7d9d0dfbf0ef72ae294c5b Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Fri, 4 Sep 2026 15:44:06 -0700 Subject: [PATCH 054/107] Add native plugin EP latency measurement setup and environment findings Records the setup for the native built-in vs shared-library plugin A/B, which is blocked: the machine's only f16-capable GPU (RTX 5060 Ti) is in a driver-failure state (Code 31), leaving a Quadro P620 that lacks shader-f16. Also records the control result that the resulting f16 failure reproduces identically on the built-in EP path, so it is not caused by the plugin boundary. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../native_plugin_latency_measurements.md | 188 ++++++++++++++++++ 1 file changed, 188 insertions(+) create mode 100644 docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md new file mode 100644 index 0000000000000..81d692dcf8e07 --- /dev/null +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md @@ -0,0 +1,188 @@ +# Native plugin EP latency measurements + +Tracks the native (shared-library) half of this workstream's latency criterion: + +> Inference latency stays within an accepted tolerance of the same baselines, for both static plugin registration and +> the native shared-library plugin. + +The static-plugin (wasm) half is already measured, in +[the work log](static_plugin_ep_work_log.md#measured-cost-of-the-plugin-path-on-the-web-build). This document covers +the native `--use_webgpu shared_lib` plugin against a built-in `--use_webgpu` baseline. + +**Status: not yet measured.** The measurement setup is complete and is recorded below, but the machine's f16-capable +GPU is currently in a driver-failure state, so no numbers have been taken. The sections below record the setup, the +environment finding, and one useful negative result, so that the run can be reproduced once the GPU is repaired. + +Commit under test: `873f221786`. + +## The measurement machine has no usable f16 GPU + +The box has three display adapters. Only one of them can run WebGPU compute at all: + +| adapter | PnP status | notes | +| --- | --- | --- | +| Microsoft Remote Display Adapter | `Unknown` | remote-session indirect display, not a compute device | +| NVIDIA Quadro P620 | `OK` | Pascal (GP107) — **no native 16-bit shader ops** | +| NVIDIA GeForce RTX 5060 Ti | **`Error`** | **Code 31 / `CM_PROB_FAILED_ADD`** — driver failed to load | + +`nvidia-smi -L` lists only the P620, and a running `onnxruntime_perf_test` was confirmed on it (86% utilization, the +process visible under GPU 0). So **every native run taken on this machine today executed on the Quadro P620**, not on +the RTX 5060 Ti. + +The failure is recent: `nvlddmkm` errors (event IDs 153 and 14) were logged at 02:01-02:03 on 2026-09-04, immediately +before that morning's 02:05 boot. Two different NVIDIA driver versions are installed side by side — `32.0.15.8267` for +the P620 and `32.0.16.1062` for the 5060 Ti — which is the usual cause of this state. Measurements recorded in earlier +sessions predate this boot and were most likely taken on a working RTX 5060 Ti. + +### Consequence: the fp16/int4 LLM case cannot run here + +Pascal does not advertise `shader-f16`, so any model needing f16 fails at the first node: + +``` +Program GatherBlockQuantized requires f16 but the device does not support it. +``` + +raised by `ORT_RETURN_IF_NOT(webgpu_context_.DeviceHasFeature(wgpu::FeatureName::ShaderF16), ...)` in +`onnxruntime/core/providers/webgpu/shader_helper.cc:421`. `WebGpuContext` only requests `ShaderF16` when the adapter +advertises it (`webgpu_context.cc:793-816`), so this is an adapter capability limit, not a configuration mistake. + +## Negative result: the f16 failure is not plugin-specific + +Worth recording, because a failure that shows up first on the plugin path invites the assumption that the plugin +boundary caused it. It did not. The same model was run on both build shapes: + +| build | EP path | result | +| --- | --- | --- | +| `build/native_plugin` | plugin (`--plugin_ep_libs`) | fails: `GatherBlockQuantized requires f16` | +| `build/int` | built-in (`-e webgpu`) | **fails identically** | + +Both arms reach the same `GetAvailableRequiredFeatures()` call on a single shared code path +(`webgpu_context.cc:157`), which is why they agree. Three mitigations were tried on the built-in arm and all failed +the same way: + +| mitigation | result | +| --- | --- | +| `-C "ep.webgpuexecutionprovider.powerPreference\|high-performance"` | still fails | +| `-C "ep.webgpuexecutionprovider.dawnBackendType\|Vulkan"` | still fails | +| verify DXC present (`dxcompiler.dll`, `dxil.dll`, `DAWN_USE_BUILT_DXC=ON`) | present in both builds; not the cause | + +The `use_dxc` toggle is already passed unconditionally by `GetEnabledAdapterToggles()` +(`webgpu_context.cc:734-748`), so DXC was never missing. + +### There is no adapter-selection knob + +`kDeviceId` (`ep.webgpuexecutionprovider.deviceId`) looks like an adapter index but is not one. It is parsed into +`WebGpuContextConfig::context_id` (`webgpu_provider_factory.cc:155-162`) and used only for context caching and +cross-device copy checks. ORT exposes no way to pick a physical adapter, so on a multi-GPU box the adapter is +whatever Dawn selects. + +**This matters for the A/B itself, not just for f16.** Record the adapter actually used alongside any numbers. If the +two arms could land on different adapters the comparison is meaningless. `--list_ep_devices` does not help — it +reports vendor `Microsoft` with no GPU identity. + +## Measurement setup + +### The matched build pair + +Two Ninja build directories from the same commit, differing only in the EP flag: + +``` +python tools\ci_build\build.py --config Release --build_dir D:\source\onnxruntime_4\build\nat_builtin ^ + --cmake_generator Ninja --build_shared_lib --skip_submodule_sync --parallel --update --build --skip_tests ^ + --target onnxruntime_perf_test --use_webgpu +``` + +and the same command with `--build_dir ...\build\nat_plugin --use_webgpu shared_lib`. + +Constraints that are easy to get wrong: + +- **`--build_shared_lib` is required on both arms.** The plugin DLL resolves `OrtGetApiBase` from the host, so a + statically linked host executable cannot load it. +- **Do not disable unit tests.** `onnxruntime_perf_test` is defined in `onnxruntime_unittests.cmake`; `--skip_tests` + only skips *running* them, which is what is wanted. +- **The plugin arm needs a second build step**, `ninja onnxruntime_providers_webgpu`, because perf_test `dlopen`s the + plugin and `--target onnxruntime_perf_test` alone does not produce the DLL. +- `build/int` is **not** usable as the baseline: it differs in four ways at once (RelWithDebInfo vs Release, Visual + Studio vs Ninja generator, `BUILD_SHARED_LIB=OFF` vs `ON`, and `ENABLE_DAWN_BACKEND_VULKAN=ON` vs `OFF`). It is + fine as a qualitative control, as used for the f16 result above, but not for timing. + +**Fairness control:** diff the two `CMakeCache.txt` files and confirm that every delta is an EP-path option. This is +the check that makes the comparison trustworthy; do not skip it on the assumption that the build commands matched. + +### Invocations + +Built-in arm: + +``` +onnxruntime_perf_test.exe -e webgpu -m times -r \model.onnx +``` + +Plugin arm: + +``` +onnxruntime_perf_test.exe --plugin_ep_libs "WebGPU|onnxruntime_providers_webgpu.dll" ^ + --plugin_eps WebGpuExecutionProvider -m times -r \model.onnx +``` + +- The EP name is **`WebGpuExecutionProvider`**, not `WebGPU`. The registration name before the `|` is arbitrary. +- perf_test **rejects a model directory** — pass the `.onnx` file. It picks up `test_data_set_0` from that directory. +- `-i` is ignored for WebGPU: `-e webgpu` hardcodes `AppendExecutionProvider("WebGPU", {})` + (`ort_test_session.cc:608-613`). Use **`-C "ep.webgpuexecutionprovider.|"`**, which routes through + `ConfigOptions` and therefore applies **identically on both arms** — exactly what an A/B needs. The available keys + are in `webgpu_provider_options.h`. + +Methodology: interleaved rounds, taking the median of each round's minimum — the same procedure used for the wasm +numbers, so the two sets are directly relatable. + +Note that perf_test **cannot drive a *statically* linked plugin EP**: both `common_utils.cc:93-99` and +`ort_test_session.cc:111-112` gate on `registered_plugin_eps`, which is populated only by `--plugin_ep_libs`. That is +moot for this dynamic-library A/B, but it is a real gap if static-plugin native perf is ever needed. + +### Models + +`bench_compute` (16 nodes) and `bench_dispatch` (300 nodes) are fp32 and run on the P620 today. `bench_dispatch` is +the dispatch-overhead-sensitive case and is where the wasm A/B found the per-kernel cost. + +A Qwen3.5-0.8B int4 case was also prepared, to check whether the per-kernel overhead is visible on a real LLM. It is +blocked on f16 and has not run. + +#### Why the Qwen inputs had to be generated by hand + +perf_test's `-I` flag cannot produce valid inputs for this model. `InitializeTensorWithSeed` +(`ort_test_session.cc:1020-1040`) randomizes only `float` and `int8_t`/`uint8_t`; every other type falls through to +`random_init = false` and prints "this type of data won't be random initialized". `input_ids` and `position_ids` are +`int64` and would be left **uninitialized**, indexing far outside the 248320-entry embedding table. Unspecified free +dimensions also default to 1 (`:1135-1137`), which would give a nonsense shape regardless. + +So a real `test_data_set_0` was generated instead. Tensor **names** are set on every `TensorProto`, which makes the +lexicographic `input_10.pb` < `input_2.pb` file ordering harmless — `TestCase.cc:621-624` prefers the name over the +positional index whenever the name is non-empty. + +The model is hybrid-attention (`qwen3_5_text`: 24 layers, vocab 248320, hidden 1024, 8 heads, 2 KV heads, +head size 256), so its 51 inputs are not uniform: + +| input | index | type | shape | +| --- | --- | --- | --- | +| `input_ids` | 0 | int64 | `[B, S]` | +| `attention_mask` | 1 | int64 | `[B, T]` | +| `past_key_values.{L}.key`, L in {3,7,11,15,19,23} | 2-7 | float16 | `[B, 2, P, 256]` | +| `past_key_values.{L}.value`, same L | 8-13 | float16 | `[B, 2, P, 256]` | +| `position_ids` | 14 | int64 | `[B, S]` | +| `past_key_values.{L}.conv_state`, the other 18 layers | 15-50 | float16 | `[B, 6144, 3]` | +| `past_key_values.{L}.recurrent_state`, same 18 layers | 15-50 | float16 | `[B, 16, 128, 128]` | + +Only 6 of the 24 layers carry a KV cache; the remaining 18 are linear-attention layers with convolution and +recurrent state. Two scenarios were materialized under `D:\test\qwen_perf`: `decode` (B=1, S=1, P=128, T=129, the +dispatch-sensitive case) and `prefill` (B=1, S=128, P=0, T=128, a compute-bound control). + +`genai_config.json` sets `past_present_share_buffer: true` and WebGPU options `enableGraphCapture: "0"` / +`validationMode: "basic"`. perf_test does not apply these, so pass them with `-C` if the comparison should match how +GenAI runs the model. + +## Results + +Pending. Re-measure once an f16-capable adapter is available, and replace the P620-era observations above with +numbers taken on the repaired GPU. + +Record with the results: the adapter actually used, the commit SHA, both build command lines, the `CMakeCache.txt` +delta, and the raw per-round minimums rather than only the medians. From 2a69c35a64eafc28c09de43745b01d088d24fb0d Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Fri, 4 Sep 2026 16:12:26 -0700 Subject: [PATCH 055/107] Update native latency doc: f16 GPU restored, measurement unblocked The RTX 5060 Ti driver was reinstalled, so the f16 blocker is resolved and both Qwen scenarios now run end to end. Reframes the environment section around the current GPU and keeps the earlier wrong-GPU episode as a cautionary note. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../native_plugin_latency_measurements.md | 94 ++++++++----------- 1 file changed, 39 insertions(+), 55 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md index 81d692dcf8e07..494f356d132d0 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md @@ -9,65 +9,51 @@ The static-plugin (wasm) half is already measured, in [the work log](static_plugin_ep_work_log.md#measured-cost-of-the-plugin-path-on-the-web-build). This document covers the native `--use_webgpu shared_lib` plugin against a built-in `--use_webgpu` baseline. -**Status: not yet measured.** The measurement setup is complete and is recorded below, but the machine's f16-capable -GPU is currently in a driver-failure state, so no numbers have been taken. The sections below record the setup, the -environment finding, and one useful negative result, so that the run can be reproduced once the GPU is repaired. - Commit under test: `873f221786`. -## The measurement machine has no usable f16 GPU - -The box has three display adapters. Only one of them can run WebGPU compute at all: +## Measurement environment -| adapter | PnP status | notes | +| adapter | PnP status | role | | --- | --- | --- | -| Microsoft Remote Display Adapter | `Unknown` | remote-session indirect display, not a compute device | -| NVIDIA Quadro P620 | `OK` | Pascal (GP107) — **no native 16-bit shader ops** | -| NVIDIA GeForce RTX 5060 Ti | **`Error`** | **Code 31 / `CM_PROB_FAILED_ADD`** — driver failed to load | +| NVIDIA GeForce RTX 5060 Ti | `OK`, driver `32.0.16.1656` (616.56), 16 GB | **the measurement device** | +| NVIDIA Quadro P620 | `Error` | Pascal, stranded on the old driver — see below | +| Microsoft Remote Display Adapter | `OK` | remote-session indirect display, not a compute device | + +`nvidia-smi -L` reports the RTX 5060 Ti as the only NVIDIA GPU. This is a convenient property for an A/B: with a +single real GPU adapter present there is no ambiguity about which device either arm ran on. -`nvidia-smi -L` lists only the P620, and a running `onnxruntime_perf_test` was confirmed on it (86% utilization, the -process visible under GPU 0). So **every native run taken on this machine today executed on the Quadro P620**, not on -the RTX 5060 Ti. +The P620 is unusable because NVIDIA ended Maxwell/Pascal/Volta support after the R570 driver branch, so the installed +616.56 package contains no kernel driver that will bind to it. Only one NVIDIA kernel driver loads at a time, so +installing a driver new enough for a Blackwell part necessarily drops the Pascal part. This is expected, not a +misconfiguration. -The failure is recent: `nvlddmkm` errors (event IDs 153 and 14) were logged at 02:01-02:03 on 2026-09-04, immediately -before that morning's 02:05 boot. Two different NVIDIA driver versions are installed side by side — `32.0.15.8267` for -the P620 and `32.0.16.1062` for the 5060 Ti — which is the usual cause of this state. Measurements recorded in earlier -sessions predate this boot and were most likely taken on a working RTX 5060 Ti. +### Earlier episode: measurements taken on the wrong GPU -### Consequence: the fp16/int4 LLM case cannot run here +Recorded because it nearly corrupted this document's results, and because the same trap applies to anyone reusing +this machine. -Pascal does not advertise `shader-f16`, so any model needing f16 fails at the first node: +For part of 2026-09-04 the situation was **inverted**: the RTX 5060 Ti was in a driver-failure state +(`CM_PROB_FAILED_ADD`, Code 31) while the P620 was healthy, so every WebGPU run silently landed on the P620. Pascal +does not advertise `shader-f16`, so f16 models failed at the first node with ``` Program GatherBlockQuantized requires f16 but the device does not support it. ``` -raised by `ORT_RETURN_IF_NOT(webgpu_context_.DeviceHasFeature(wgpu::FeatureName::ShaderF16), ...)` in +from `ORT_RETURN_IF_NOT(webgpu_context_.DeviceHasFeature(wgpu::FeatureName::ShaderF16), ...)` in `onnxruntime/core/providers/webgpu/shader_helper.cc:421`. `WebGpuContext` only requests `ShaderF16` when the adapter -advertises it (`webgpu_context.cc:793-816`), so this is an adapter capability limit, not a configuration mistake. +advertises it (`webgpu_context.cc:793-816`), so this was an adapter capability limit rather than a configuration +mistake. Reinstalling the driver resolved it, and both Qwen scenarios then ran clean. -## Negative result: the f16 failure is not plugin-specific +Two lessons worth keeping: -Worth recording, because a failure that shows up first on the plugin path invites the assumption that the plugin -boundary caused it. It did not. The same model was run on both build shapes: - -| build | EP path | result | -| --- | --- | --- | -| `build/native_plugin` | plugin (`--plugin_ep_libs`) | fails: `GatherBlockQuantized requires f16` | -| `build/int` | built-in (`-e webgpu`) | **fails identically** | - -Both arms reach the same `GetAvailableRequiredFeatures()` call on a single shared code path -(`webgpu_context.cc:157`), which is why they agree. Three mitigations were tried on the built-in arm and all failed -the same way: - -| mitigation | result | -| --- | --- | -| `-C "ep.webgpuexecutionprovider.powerPreference\|high-performance"` | still fails | -| `-C "ep.webgpuexecutionprovider.dawnBackendType\|Vulkan"` | still fails | -| verify DXC present (`dxcompiler.dll`, `dxil.dll`, `DAWN_USE_BUILT_DXC=ON`) | present in both builds; not the cause | - -The `use_dxc` toggle is already passed unconditionally by `GetEnabledAdapterToggles()` -(`webgpu_context.cc:734-748`), so DXC was never missing. +- **A failure seen first on the plugin path is not necessarily caused by the plugin path.** The f16 failure was + reproduced identically on the built-in arm (`build/int`, `-e webgpu`), which is what identified it as + environmental. Both arms reach the same `GetAvailableRequiredFeatures()` call (`webgpu_context.cc:157`), so they + agree by construction. Always run the built-in control before attributing anything to the plugin boundary. +- **Timing runs do not announce which GPU they used.** Nothing in ORT's output names the adapter, so a machine with + more than one GPU can quietly produce numbers from the wrong one. Confirm the device externally — `nvidia-smi` + during a long run shows the process against a specific GPU — and record it alongside the numbers. ### There is no adapter-selection knob @@ -76,9 +62,10 @@ The `use_dxc` toggle is already passed unconditionally by `GetEnabledAdapterTogg cross-device copy checks. ORT exposes no way to pick a physical adapter, so on a multi-GPU box the adapter is whatever Dawn selects. -**This matters for the A/B itself, not just for f16.** Record the adapter actually used alongside any numbers. If the -two arms could land on different adapters the comparison is meaningless. `--list_ep_devices` does not help — it -reports vendor `Microsoft` with no GPU identity. +**This matters for the A/B itself, not just for f16.** If the two arms could land on different adapters the +comparison is meaningless, and neither ORT nor perf_test will tell you — `--list_ep_devices` reports vendor +`Microsoft` with no GPU identity. On this machine the point is currently moot, since only one real GPU adapter is +present, but record the adapter alongside any numbers regardless. ## Measurement setup @@ -140,11 +127,11 @@ moot for this dynamic-library A/B, but it is a real gap if static-plugin native ### Models -`bench_compute` (16 nodes) and `bench_dispatch` (300 nodes) are fp32 and run on the P620 today. `bench_dispatch` is -the dispatch-overhead-sensitive case and is where the wasm A/B found the per-kernel cost. +`bench_compute` (16 nodes) and `bench_dispatch` (300 nodes) are fp32. `bench_dispatch` is the +dispatch-overhead-sensitive case, and is where the wasm A/B found the per-kernel cost. -A Qwen3.5-0.8B int4 case was also prepared, to check whether the per-kernel overhead is visible on a real LLM. It is -blocked on f16 and has not run. +A Qwen3.5-0.8B int4 case is also included, to check whether the per-kernel overhead is visible on a real LLM. Both +its scenarios have been confirmed to run end to end on the RTX 5060 Ti. #### Why the Qwen inputs had to be generated by hand @@ -181,8 +168,5 @@ GenAI runs the model. ## Results -Pending. Re-measure once an f16-capable adapter is available, and replace the P620-era observations above with -numbers taken on the repaired GPU. - -Record with the results: the adapter actually used, the commit SHA, both build command lines, the `CMakeCache.txt` -delta, and the raw per-round minimums rather than only the medians. +Pending — the matched build pair is still building. Record with the results: the adapter used, the commit SHA, both +build command lines, the `CMakeCache.txt` delta, and the raw per-round minimums rather than only the medians. From ebe31d54772113c6418f31f88f0dce71f6b11363 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Fri, 4 Sep 2026 17:43:12 -0700 Subject: [PATCH 056/107] Document build fairness controls for the native latency A/B An interrupted CMake configure left CMAKE_CXX_FLAGS_RELEASE empty in one build directory. A later --update reused the damaged cache and silently produced an unoptimized binary that would have skewed the comparison toward the plugin. Adds the build.ninja optimization-flag check that caught it, and notes that perf_test's exit code is unreliable because of the shutdown leak checker. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../native_plugin_latency_measurements.md | 42 ++++++++++++++++++- 1 file changed, 40 insertions(+), 2 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md index 494f356d132d0..fa1a04c4b0c9f 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md @@ -93,8 +93,42 @@ Constraints that are easy to get wrong: Studio vs Ninja generator, `BUILD_SHARED_LIB=OFF` vs `ON`, and `ENABLE_DAWN_BACKEND_VULKAN=ON` vs `OFF`). It is fine as a qualitative control, as used for the f16 result above, but not for timing. -**Fairness control:** diff the two `CMakeCache.txt` files and confirm that every delta is an EP-path option. This is -the check that makes the comparison trustworthy; do not skip it on the assumption that the build commands matched. +### Fairness control: verify the two builds differ only in the EP + +Identical `build.py` command lines do **not** guarantee identical builds, so verify the generated build rather than +trusting the invocation. + +1. Diff the two `CMakeCache.txt` files and confirm every delta is an EP-path option. Normalize the build-directory + name first, or the path embedded in most entries will swamp the diff. +2. Confirm the optimization flags actually reached the compiler, by counting them in the generated `build.ninja`: + +```powershell +$bn = 'D:\source\onnxruntime_4\build\\Release\build.ninja' +foreach ($f in '/O2','/Ob2','/DNDEBUG') { + "$f = " + (Select-String -Path $bn -Pattern $f -SimpleMatch -AllMatches | Measure-Object).Count +} +``` + +Both arms should report several thousand of each. **Zero means an unoptimized build**: MSVC defaults to `/Od` when +no `/O` flag is given, and a missing `/DNDEBUG` also leaves asserts enabled. + +This is not hypothetical — step 2 caught exactly that failure here. A first `nat_builtin` configure was interrupted +part-way through and left `CMAKE_CXX_FLAGS_RELEASE` **empty** in its cache. A later `build.py --update` reused the +damaged cache rather than regenerating it, and the build then completed with no error, producing a working but +**unoptimized** `onnxruntime_perf_test.exe`. The two arms' compile flags were otherwise character-identical: + +``` +nat_builtin: ... -DEIGEN_HAS_C99_MATH -std:c++20 -MD -Zi /GR /W4 ... +nat_plugin: ... -DEIGEN_HAS_C99_MATH /O2 /Ob2 /DNDEBUG -std:c++20 -MD -Zi /GR /W4 ... +``` + +Had this gone unnoticed it would have produced a large, entirely bogus result in the plugin's favour. The fix is to +**delete the build directory and configure from scratch** — repairing a cache in an unknown state is not worth the +risk. A corrupted cache also shows a secondary tell: stray `CMAKE_ADDR2LINE`, `CMAKE_NM`, `CMAKE_OBJCOPY`, +`CMAKE_OBJDUMP`, `CMAKE_READELF`, `CMAKE_STRIP` and `CMAKE_TAPI` entries that the healthy cache does not have. + +**Generalization: if a build directory was ever interrupted mid-configure, do not measure with it.** Interrupted +CMake configures fail silently in a way that survives `--update` and still produces a runnable binary. ### Invocations @@ -117,6 +151,10 @@ onnxruntime_perf_test.exe --plugin_ep_libs "WebGPU|onnxruntime_providers_webgpu. (`ort_test_session.cc:608-613`). Use **`-C "ep.webgpuexecutionprovider.|"`**, which routes through `ConfigOptions` and therefore applies **identically on both arms** — exactly what an A/B needs. The available keys are in `webgpu_provider_options.h`. +- **Do not gate on the process exit code.** perf_test returns non-zero even on a fully successful run, because ORT's + memory-leak checker reports a handful of 16-byte CRT static-initializer allocations (`initterm`) at shutdown. The + report is emitted after timing and affects both arms identically, so it does not bias the comparison — but scripts + should parse `Min Latency` from stdout rather than trusting the exit status. Methodology: interleaved rounds, taking the median of each round's minimum — the same procedure used for the wasm numbers, so the two sets are directly relatable. From 1cfea6f53ff2b75b3ae84c8394ff5ebbb437b6c6 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Tue, 8 Sep 2026 13:36:36 -0700 Subject: [PATCH 057/107] Record native built-in vs plugin EP latency A/B results Adds the measured results section and the harness script. The plugin EP is consistently slower: ~8% on bench_dispatch (300 nodes) and 4-6% on Qwen3.5-0.8B, which fits a model of roughly 67us per Run plus 1.2us per node. That is ~3-4x cheaper per node than the wasm static-plugin boundary's ~4.8us/node. Controls, since the result is small enough to be confused with noise: - the two builds differ by exactly one CMakeCache entry (onnxruntime_USE_EP_API_ADAPTERS), and both are optimized - the plugin build rejects -e webgpu, so a successful run can only have come from onnxruntime_providers_webgpu.dll - running both binaries on the CPU EP, where no plugin is involved, gives a ratio of minima of 1.001, so the WebGPU delta is the plugin boundary rather than a build difference Raw per-round minimums are recorded rather than medians alone, because the two Qwen rows are noise-dominated (per-round deltas swing -4.6 to +9.3 ms) and only bench_dispatch has a spread tight enough to resolve the effect cleanly. --- .../native_plugin_latency_measurements.md | 115 +++++++++++++++++- .../run_native_ab.ps1 | 78 ++++++++++++ 2 files changed, 191 insertions(+), 2 deletions(-) create mode 100644 docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/run_native_ab.ps1 diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md index fa1a04c4b0c9f..d3715eb0b8f02 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md @@ -206,5 +206,116 @@ GenAI runs the model. ## Results -Pending — the matched build pair is still building. Record with the results: the adapter used, the commit SHA, both -build command lines, the `CMakeCache.txt` delta, and the raw per-round minimums rather than only the medians. +Measured on the RTX 5060 Ti, on an otherwise idle machine. The two `build.py` configures differ by exactly one +generator flag, `-Donnxruntime_USE_EP_API_ADAPTERS=ON` on the plugin arm. + +### Checks performed before measuring + +| check | result | +| --- | --- | +| Normalized `CMakeCache.txt` diff | **2 lines, one key**: `onnxruntime_USE_EP_API_ADAPTERS` `OFF` vs `ON` | +| `/O2` / `/Ob2` / `/DNDEBUG` in `build.ninja` | 4184 built-in, 4183 plugin — off-by-one is WebGPU built in-tree | +| Binaries current with the tree | every commit since `873f221786` is doc-only, so the code under test is HEAD's | + +### The plugin arm really is going through the plugin + +A latency number is meaningless if the "plugin" arm quietly fell back to something else, so this was established by +making the alternatives fail rather than by reading the code: + +| invocation on the plugin build | outcome | +| --- | --- | +| `-e webgpu` (built-in path) | fails: "WebGPU execution provider is not supported in this build" | +| `--plugin_eps BogusEpName` | fails: "[Plugin EP]: No matching EP devices found." (`common_utils.cc:171`) | +| `--plugin_eps WebGpuExecutionProvider` | runs | + +The first line is the important one: WebGPU is genuinely absent from this build's `onnxruntime.dll`, so a run that +succeeds can only have come from `onnxruntime_providers_webgpu.dll`. The second confirms the EP-name lookup is real +and not silently permissive. + +### Control: the two binaries are equivalent when no plugin is involved + +Matching build flags still do not prove the two executables perform alike. Running both on the **CPU EP**, where +neither arm loads a plugin, isolates the plugin boundary from any residual build difference. + +Run with default threading this control was actively misleading — `bench_compute` came out **24% faster** on the +plugin executable (4.61 ms vs 3.50 ms), which is not a real effect in any direction and simply shows how noisy the +multi-threaded CPU EP is here. Repeating it single-threaded (`-x 1 -y 1`, 10 rounds) removes the thread-pool +variance: + +| `bench_compute`, CPU EP, single-threaded | min | median | max | +| --- | --- | --- | --- | +| built-in build | 20.011 ms | 20.285 ms | 20.734 ms | +| plugin build | 20.023 ms | 20.499 ms | 20.848 ms | + +Ratio of minima **1.001**, medians within 1%. The two executables are performance-equivalent absent a plugin, so the +WebGPU differences below are attributable to the plugin boundary rather than to the builds. + +### WebGPU A/B + +8 interleaved rounds; `-r 30` for the bench models, `-r 20` for Qwen. "median" is the median of the per-round +minimums, matching the wasm procedure; "best" is the single fastest round, which is the most noise-resistant +statistic available. + +| model | nodes | built-in median | plugin median | delta | ratio | best-of ratio | rounds plugin slower | +| --- | --- | --- | --- | --- | --- | --- | --- | +| `bench_compute` | 16 | 1.662 ms | 1.917 ms | +0.255 ms | 1.153 | 1.054 | 8/8 | +| `bench_dispatch` | 300 | 5.118 ms | 5.562 ms | +0.444 ms | 1.087 | 1.084 | 8/8 | +| Qwen decode | 1043 | 34.95 ms | 36.95 ms | +1.99 ms | 1.057 | 1.092 | 6/8 | +| Qwen prefill | 1043 | 47.66 ms | 49.57 ms | +1.92 ms | 1.040 | 1.064 | 6/8 | + +**The plugin EP is consistently slower.** Raw per-round minimums, in milliseconds, in round order: + +| model | arm | per-round minimums | +| --- | --- | --- | +| `bench_compute` | built-in | 1.653, 1.626, 1.686, 1.661, 1.895, 1.877, 1.662, 1.614 | +| `bench_compute` | plugin | 1.700, 1.862, 2.055, 1.702, 1.971, 2.189, 1.701, 2.345 | +| `bench_dispatch` | built-in | 5.170, 5.131, 5.105, 5.158, 5.112, 5.124, 5.111, 5.104 | +| `bench_dispatch` | plugin | 5.577, 5.544, 5.623, 5.584, 5.543, 5.546, 6.091, 5.533 | +| Qwen decode | built-in | 31.34, 34.90, 40.56, 36.13, 35.01, 38.00, 30.57, 34.63 | +| Qwen decode | plugin | 35.32, 40.86, 43.65, 34.24, 36.33, 33.40, 39.88, 37.56 | +| Qwen prefill | built-in | 43.05, 49.42, 44.24, 50.92, 47.75, 47.56, 46.17, 48.23 | +| Qwen prefill | plugin | 49.99, 49.56, 50.59, 52.32, 48.80, 45.80, 49.58, 45.89 | + +### How much of this is signal + +The four rows are not equally trustworthy, and the medians alone hide that: + +- **`bench_dispatch` is the reliable result.** The built-in arm spans 5.104–5.170 ms — a 66 µs range — and the + per-round delta is +0.41 to +0.52 ms in seven of eight rounds (the eighth, +0.98 ms, is a visible outlier). An + effect of +0.43 ms against that spread is unambiguous. +- **`bench_compute` is noisier than its median suggests.** Per-round deltas range from +0.039 to +0.731 ms. The + plugin arm's slow rounds inflate the median to 1.153; the best-of ratio of 1.054 is the more defensible figure. +- **Both Qwen rows are noise-dominated.** Per-round deltas swing from −4.6 to +9.3 ms, and the sign flips in 2 of 8 + rounds. The direction agrees with the microbenchmarks on both the median and the best-of statistic, but these + numbers should be read as "consistent with a few percent" and not as a measurement of it. + +### Mechanism + +Fitting the two microbenchmarks' best-of deltas as a fixed per-`Run` cost plus a per-node cost gives roughly +**67 µs per `Run` + 1.2 µs per node**. That is a two-point fit and should be treated as an order-of-magnitude model +only, but it does predict outside its fitting range: for Qwen's 1043 nodes it gives ≈1.3 ms, against an observed +median delta of ≈1.9–2.0 ms — the right magnitude, and under-predicting by about 1.5x. Per-node dispatch cost is +therefore a plausible dominant term, which is what the shape of the boundary would predict: each kernel dispatch +crosses the `OrtEp` C API adapter into a separate DLL instead of making an in-module C++ virtual call. + +Compared with the wasm static-plugin A/B, which found ≈4.8 µs/node +([work log](static_plugin_ep_work_log.md#measured-cost-of-the-plugin-path-on-the-web-build)), the native +shared-library boundary costs roughly **1.2–1.5 µs/node**, some 3–4x less per node. + +### Assessment against the criterion + +A per-node cost means the overhead is a function of graph size, not a fixed tax, so no single percentage +characterizes it. On the real-model cases it lands at **4–6%**, and it grows as models get more numerous, smaller +kernels. Whether that is "within an accepted tolerance" is a judgement call that has not yet been made — the +tolerance has never been quantified, and it should be, since these results show the answer is not "zero overhead". + +Two caveats on scope: + +- Only steady-state `Min Latency` was compared. Session-creation cost, where the plugin must additionally discover + and load the DLL, was not measured and is likely worse in relative terms. +- Both arms ran with default WebGPU options. `genai_config.json` uses `enableGraphCapture: "0"` and + `validationMode: "basic"`; graph capture in particular could change the per-dispatch picture substantially, and is + worth a follow-up since it targets exactly the cost identified here. + +Reproduce with `run_native_ab.ps1` (parses `Min Latency`, ignores the exit code); raw rows land in +`native_ab_raw.csv`. diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/run_native_ab.ps1 b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/run_native_ab.ps1 new file mode 100644 index 0000000000000..fb42062576ab1 --- /dev/null +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/run_native_ab.ps1 @@ -0,0 +1,78 @@ +# Interleaved A/B: built-in WebGPU EP vs shared-library plugin EP. +# Parses "Min Latency" from stdout; perf_test's exit code is unreliable because the +# shutdown leak checker reports benign CRT static-init allocations. +param( + [int]$Rounds = 8, + [int]$Reps = 30, + [int]$QwenReps = 20, + [string]$OutCsv = 'D:\test\native_ab_raw.csv' +) + +$builtinExe = 'D:\source\onnxruntime_4\build\nat_builtin\Release\onnxruntime_perf_test.exe' +$pluginDir = 'D:\source\onnxruntime_4\build\nat_plugin\Release' +$pluginExe = "$pluginDir\onnxruntime_perf_test.exe" + +$models = [ordered]@{ + 'bench_compute' = 'D:\source\onnxruntime_4\js\web\test\data\bench\bench_compute\model.onnx' + 'bench_dispatch' = 'D:\source\onnxruntime_4\js\web\test\data\bench\bench_dispatch\model.onnx' + 'qwen_decode' = 'D:\test\qwen_perf\decode\model.onnx' + 'qwen_prefill' = 'D:\test\qwen_perf\prefill\model.onnx' +} + +function Get-MinLatency([string]$text) { + $m = [regex]::Match($text, 'Min Latency:\s*([0-9.eE+-]+)\s*s') + if ($m.Success) { return [double]$m.Groups[1].Value } + return $null +} + +function Invoke-Arm([string]$arm, [string]$model, [int]$reps) { + if ($arm -eq 'builtin') { + $out = & $builtinExe -e webgpu -m times -r $reps $model 2>&1 | Out-String + } else { + Push-Location $pluginDir + try { + $out = & $pluginExe --plugin_ep_libs "WebGPU|onnxruntime_providers_webgpu.dll" ` + --plugin_eps WebGpuExecutionProvider -m times -r $reps $model 2>&1 | Out-String + } finally { Pop-Location } + } + return (Get-MinLatency $out) +} + +function Get-Median([double[]]$v) { + $s = $v | Sort-Object + $n = $s.Count + if ($n -eq 0) { return $null } + if ($n % 2 -eq 1) { return $s[[int](($n-1)/2)] } + return ($s[$n/2 - 1] + $s[$n/2]) / 2 +} + +$rows = @() +foreach ($r in 1..$Rounds) { + foreach ($name in $models.Keys) { + $reps = if ($name -like 'qwen*') { $QwenReps } else { $Reps } + # Interleave within the round so slow drift affects both arms equally. + $b = Invoke-Arm 'builtin' $models[$name] $reps + $p = Invoke-Arm 'plugin' $models[$name] $reps + $rows += [pscustomobject]@{ + round = $r; model = $name + builtin_ms = if ($b -ne $null) { [math]::Round($b*1000,4) } else { $null } + plugin_ms = if ($p -ne $null) { [math]::Round($p*1000,4) } else { $null } + } + Write-Host ("round {0} {1,-15} builtin={2,9:N4} ms plugin={3,9:N4} ms" -f $r,$name,($b*1000),($p*1000)) + } +} + +$rows | Export-Csv -NoTypeInformation -Path $OutCsv +Write-Host "`n=== raw rows written to $OutCsv ===" +Write-Host "`n=== SUMMARY (median of per-round minimums) ===" +Write-Host ("{0,-15} {1,12} {2,12} {3,10} {4,9} {5,13}" -f ` + 'model','builtin_ms','plugin_ms','delta_ms','ratio','plugin_slower') +foreach ($name in $models.Keys) { + $sub = $rows | Where-Object { $_.model -eq $name -and $_.builtin_ms -ne $null -and $_.plugin_ms -ne $null } + if (-not $sub) { Write-Host ("{0,-15} NO DATA" -f $name); continue } + $mb = Get-Median ([double[]]($sub.builtin_ms)) + $mp = Get-Median ([double[]]($sub.plugin_ms)) + $slower = ($sub | Where-Object { $_.plugin_ms -gt $_.builtin_ms }).Count + Write-Host ("{0,-15} {1,12:N4} {2,12:N4} {3,10:N4} {4,9:N3} {5,10}/{6}" -f ` + $name, $mb, $mp, ($mp-$mb), $(if($mb -gt 0){$mp/$mb}else{0}), $slower, $sub.Count) +} From 89b5144be10351dea4e42eaa41c81b766a2994ae Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Tue, 8 Sep 2026 13:37:52 -0700 Subject: [PATCH 058/107] Log the native plugin latency A/B in the work log Records the native measurement alongside the wasm one, plus the three process failures it exposed: measuring on the wrong GPU after a driver fault, one arm silently built unoptimized by a reused damaged CMake cache, and needing a CPU-EP control to attribute the delta to the plugin boundary rather than to the builds. Adds an open item for the unquantified latency tolerance, since both plugin shapes charge per kernel node rather than a fixed amount, so no single percentage characterizes the overhead. Also corrects the status table: the ORT Web row still said 'not yet run in a browser', which the document's own 'Running it in a browser' section and open item 4 already contradict. --- .../static_plugin_ep_work_log.md | 43 ++++++++++++++++++- 1 file changed, 42 insertions(+), 1 deletion(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md index 2bbd46a1a3b97..9d987da0a18cb 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md @@ -32,7 +32,8 @@ The implementation is complete and committed. CI coverage was added for the new | Operator-level validation on a real GPU | **Done — no behavioural difference**, see [Testing](#testing) | | Linux/GCC build (`--use_webgpu static_plugin`) | **Green in CI** (run `33030694040`) after the `array-bounds` fix — see [GCC](#gcc-and-warnings-as-errors) | | Minimal build | Not yet exercised — see [Open items](#open-items) | -| Emscripten / ORT Web | **Builds and links**, plugin registration verified in the binary — see [ORT Web](#emscripten-and-ort-web). Not yet run in a browser | +| Emscripten / ORT Web | **Builds, links, and runs in a browser** on a real GPU; registration verified in the binary *and* at runtime — see [ORT Web](#emscripten-and-ort-web) | +| Latency | **Measured on both plugin shapes** — ~4.8 us/node (wasm), ~1.2 us/node (native); the accepted tolerance is still unquantified, see [Open items](#open-items) | | PR | Draft [#32395](https://github.com/microsoft/onnxruntime/pull/32395); self-review feedback addressed — see [PR review feedback](#pr-review-feedback) | ## Branch @@ -637,6 +638,36 @@ no-op. The evidence is strong for the web build, where the patched `GPUQueue.sub instrument for the claimed harm. It is weaker for native, where only timing was compared and submits were never counted. +## Native shared-library plugin measurements + +The latency criterion covers two shapes of plugin, and the wasm numbers above only settle the static one. The +native `--use_webgpu shared_lib` A/B against a built-in baseline is written up separately in +[native_plugin_latency_measurements.md](native_plugin_latency_measurements.md), with the harness in +`run_native_ab.ps1`. Summary: the plugin is slower by ~8% on `bench_dispatch` and 4-6% on Qwen3.5-0.8B, fitting +roughly **67 us per `Run` plus 1.2 us per node** — about 3-4x cheaper per node than the wasm boundary's 4.8 us. + +Three things from that exercise are worth recording here, because each cost real time and none was visible from +reading code: + +- **The first run measured the wrong GPU.** A driver failure had left the RTX 5060 Ti in PnP error state, so + every WebGPU run silently landed on a Quadro P620, which lacks `shader-f16` and could not run the int4 Qwen + model at all. Nothing in ORT's output names the adapter, and there is no adapter-selection knob, so on a + multi-GPU box the adapter has to be confirmed externally with `nvidia-smi` during a run. +- **One arm was accidentally built unoptimized.** An interrupted CMake configure left `CMAKE_CXX_FLAGS_RELEASE` + empty; a later `build.py --update` reused the damaged cache instead of regenerating it, and the build + succeeded with no error. Counting `/O2` in the generated `build.ninja` catches this; the cache alone does not + make it obvious. Had it gone unnoticed it would have produced a large bogus result favouring the plugin. +- **A CPU-EP control was needed to attribute the delta.** Matching build flags do not prove two executables + perform alike. Running both binaries on the CPU EP, where no plugin is loaded, gave a ratio of minima of + 1.001 — which is what licenses reading the WebGPU delta as plugin-boundary cost. Run with default threading + that same control was misleading, showing the plugin executable 24% *faster*; it only became usable + single-threaded. + +The relationship to the wasm result is the interesting part: both boundaries charge per kernel node, and the +native one charges less. That is consistent with the wasm cost being dominated by something inside the +`static_plugin` wasm's per-kernel C++ path rather than by the plugin indirection itself, which is what the +three-way `mismatch` run above already suggested. + ## PR review feedback Draft PR [#32395](https://github.com/microsoft/onnxruntime/pull/32395) collected eleven inline self-review @@ -812,6 +843,16 @@ RTX 5060 Ti machine, `onnxruntime_test_all --gtest_filter=InferenceSessionTests. dropped; **decision: deferred until this branch lands**, at which point the link becomes valid and the whole set can go across together. +8. **Decide the latency tolerance.** The native half of the latency criterion is now measured — see + [Native shared-library plugin measurements](#native-shared-library-plugin-measurements). Both plugin shapes + cost per kernel node rather than a fixed amount, so no single percentage characterizes the overhead: it is + ~4-6% on real models and grows with node count. The criterion says latency must stay "within an accepted + tolerance", but that tolerance has never been quantified, and these results show the answer is not "zero". + Two gaps remain in the native numbers: session-creation cost was not compared (the plugin arm must also + discover and load the DLL, so it is likely worse in relative terms), and both arms ran with default WebGPU + options — graph capture in particular targets exactly the per-dispatch cost identified, and is worth a + follow-up. + ## Environment notes Specific to the original machine, but the class of problem generalises. From a888054f8554b04ed5c0b4621238a681fd6aab42 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Tue, 8 Sep 2026 14:41:11 -0700 Subject: [PATCH 059/107] Record CPU profile root-causing the native plugin EP per-node overhead Profiling both A/B arms shows the ~1.2 us/node native plugin overhead is not boundary-crossing cost but redundant per-kernel work: the EP API adapter rebuilds an onnxruntime::Tensor from an OrtValue for every input and output of every node on every Run. Records the method (custom sampling profiler, after ORT's -p profiler and xperf both proved unusable), the pooled results, the plugin-only symbols, and the caveats. Also logs that the plugin path emits no Api profiling events, an observability regression. --- .../native_plugin_latency_measurements.md | 122 ++++++++++++++++++ .../static_plugin_ep_work_log.md | 51 ++++++-- 2 files changed, 163 insertions(+), 10 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md index d3715eb0b8f02..b396176f609df 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md @@ -319,3 +319,125 @@ Two caveats on scope: Reproduce with `run_native_ab.ps1` (parses `Min Latency`, ignores the exit code); raw rows land in `native_ab_raw.csv`. + +## Where the per-node cost goes (CPU profile) + +The 1.2 µs/node figure above is far too large to be call-transport cost — a cross-DLL indirect call through the C API +is tens of cycles, while 1.2 µs at 3.696 GHz is ≈4,400 cycles. That gap motivated profiling the two arms to find out +what the cycles are actually spent on. The answer is that they are spent on **per-kernel marshalling work that the +built-in path does not do at all**, not on crossing the boundary. + +### Method + +Both arms were relinked with `/DEBUG` to produce PDBs. This was a link-only change: `LINK_FLAGS` already carried an +explicit `/OPT:REF,ICF,LBR`, so adding `/DEBUG` does not fall back to the `/OPT:NOREF,NOICF` debug defaults, and no +source file was recompiled. Binary sizes were unchanged and the A/B ratio was re-measured at 1.087 afterwards, so the +profiled binaries are the same ones the latency numbers came from. + +Two profilers were tried and rejected before settling on a third: + +- **ORT's own `-p` profiler cannot be used for this comparison.** Instrumentation is asymmetric — the built-in arm + emits 15,300 `Api` category events and the plugin arm emits **zero**, so under `-p` the built-in arm measures + *slower* (0.772 s vs 0.644 s over 50 runs), reversing the true result. It also inflates per-node time from ~17 µs + to ~46 µs by forcing per-node synchronisation. One result did survive: **per-op median kernel time is identical on + both arms (29.0 µs for Add/Mul/Sub)**, establishing early that GPU-side work is unchanged and the cost is host-side. + That the plugin path silently loses the WebGPU EP's `Api` profiling events is itself an observability regression. +- **ETW/xperf resolved no symbols.** Traces collected fine (355k+ stack samples), but every frame rendered as + `***unknown***` — zero `Module!Func` tokens in the entire report — despite symcache files being generated for all + three PDBs, and with or without the Microsoft symbol server on the path. +- **A purpose-built sampling profiler** (`sampler.cpp`) was used instead: it launches the target, samples every thread's + instruction pointer at ~1 kHz, and symbolizes leaf addresses with dbghelp against the local PDBs. Needs no + elevation, and symbolizes ORT frames reliably. + +Runs were 6000 reps of `bench_dispatch` (300 nodes), discarding the first 3 s, with the two arms pooled over two +repetitions. Sampling did not destroy the effect being studied: the latency ratio under the profiler was 1.083, +1.081 and 1.083 across three runs, against 1.087 unsampled. + +### Result + +Samples are counted only when a thread is actually running (leaf address not in a wait stub). One sample corresponds +to roughly one sampler loop period, ~1.0–1.2 ms of thread time, so **absolute µs below are approximate to about +±20%; the relative comparisons are not affected.** + +| Scope | built-in | plugin | delta | ratio | +| --- | ---: | ---: | ---: | ---: | +| All modules, running samples/rep | 3.388 | 3.922 | +0.534 | 1.158 | +| `onnxruntime.dll` + `onnxruntime_providers_webgpu.dll` only | 1.175 | 1.441 | +0.267 | 1.227 | + +Per-module numbers are meaningless on their own here, because the WebGPU code simply *moves* out of `onnxruntime.dll` +into `onnxruntime_providers_webgpu.dll`; only the combined ORT-side total is comparable. + +The two totals cross-check the latency result. The extra CPU of +0.534 samples/rep is ≈0.53–0.64 ms/rep against a +measured extra wall latency of 0.42 ms/rep, and the ORT-side share alone (+0.267 samples/rep ≈ 0.9–1.1 µs/node) is +close to the 1.2 µs/node from the latency fit. **The overhead is host-side CPU work, and most of it is ORT-side.** + +Attributing the +0.534 samples/rep by what the code is doing: + +| Category | built-in | plugin | delta | share of delta | +| --- | ---: | ---: | ---: | ---: | +| other (mostly GPU driver, see below) | 1.752 | 1.952 | +0.199 | 37.3% | +| alloc/free | 0.573 | 0.669 | +0.096 | 17.9% | +| `Tensor`/`TensorShape` construction | 0.083 | 0.165 | +0.082 | 15.3% | +| EP API adapter | 0.002 | 0.074 | +0.072 | 13.4% | +| dawn (refcounting, error checks) | 0.495 | 0.549 | +0.054 | 10.1% | +| webgpu kernel bodies | 0.173 | 0.208 | +0.034 | 6.4% | +| ort framework | 0.128 | 0.118 | −0.011 | −2.0% | + +The `ep-adapter` and `Tensor`/`TensorShape` rows are the reproducible core: they were +0.060/+0.063 and +0.030/+0.032 +in the two individual runs, i.e. stable, and the adapter work is *new* — it is ~0.002 in the built-in arm, meaning it +does not exist there. The `dawn`, `webgpu-kernel` and `alloc/free` rows moved noticeably between the two runs and +should be read as directional only. + +These symbols appear in the plugin arm and are absent from the built-in arm: + +| Symbol | plugin samples/rep | +| --- | ---: | +| `onnxruntime::TensorShape::TensorShape` | 0.020 | +| `onnxruntime::webgpu::ComputeContext::Input` | 0.016 | +| `onnxruntime::ep::adapter::CreateTensorFromApiValue` | 0.013 | +| `Ort::GetApi` | 0.013 | +| `onnxruntime::TensorShape::Allocate` | 0.013 | +| `onnxruntime::PluginEpOpKernel::Compute` | 0.008 | +| `onnxruntime::Tensor::operator=` | 0.007 | +| `onnxruntime::ep::adapter::KernelImpl::ComputeImpl` | 0.006 | +| `OrtApis::IsTensor` | 0.006 | +| `OrtApis::GetTensorElementTypeAndShapeDataReference` | 0.005 | +| `onnxruntime::DataTypeImpl::TensorTypeFromONNXEnum` | 0.004 | + +### Mechanism + +The profile points at `ep::adapter::CreateTensorFromApiValue` +([`include/onnxruntime/ep/adapter/tensor_helper.h`](../../../../include/onnxruntime/ep/adapter/tensor_helper.h)), +which rebuilds an `onnxruntime::Tensor` from an `OrtValue` on every access. Each call makes roughly nine C API round +trips (`IsTensor`, `GetTensorElementTypeAndShapeDataReference`, `GetTensorMemoryInfo`, then `GetAllocatorName`, +`GetAllocatorType`, `GetDeviceType`, `GetMemoryType`, `GetVendorId`, `GetDeviceId`), allocates a `std::string` for +the allocator name, constructs a `TensorShape` (a heap allocation once the rank exceeds the inline capacity), looks +up `DataTypeImpl::TensorTypeFromONNXEnum`, and constructs a `Tensor`. + +The adapter `OpKernelContext` ([`op_kernel.h`](../../../../include/onnxruntime/ep/adapter/op_kernel.h)) is built +fresh per `Compute()` call and `resize()`s an input and an output `std::vector`. Its tensor cache lives only +for the duration of that one call, so the work above repeats for every input and every output of every node on every +`Run`. The built-in path does none of this: it receives `const Tensor*` directly from the executor. + +That is the answer to the original question. The overhead is **not** the cost of crossing the DLL boundary; it is +redundant per-kernel wrapper construction and its allocation traffic, which is why it is ~4,400 cycles rather than +the tens of cycles an indirect cross-module call would cost. + +### Caveats + +- Part of the delta is plausibly *consequence* rather than cause. The `other` bucket is dominated by GPU driver + modules (`nvwgf2umx.dll` +0.085, `win32u.dll` +0.017, `D3D12Core.dll` +0.014), and driver threads that poll will + accumulate samples in proportion to wall time — which the plugin arm has more of. The unambiguous causal signal is + the adapter and tensor-construction work, which is absent from the built-in arm entirely. +- Driver and `ntdll` frames symbolize to the nearest export (no PDBs available), so names such as + `RtlCreateUnicodeString+0x118f` identify a region, not the actual function. Module attribution is still sound. +- Suspend/resume sampling perturbs both arms, but equally; the preserved latency ratio is the check on this. +- This is a flat (leaf-address) profile, so it attributes self time only and does not give call trees. + +### Implication for the tolerance question + +Open item 8 in the work log asks what tolerance is acceptable. These results argue against negotiating one yet: the +dominant identified cost is redundant work in the adapter, not an intrinsic property of the plugin boundary, so it +looks reducible — for example by caching the `Tensor` wrapper across `Compute()` calls, or by avoiding the +allocator-name string and shape allocation on the per-node path. The tolerance question is better asked after that +is attempted. diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md index 9d987da0a18cb..d05288b35a8f8 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md @@ -33,7 +33,7 @@ The implementation is complete and committed. CI coverage was added for the new | Linux/GCC build (`--use_webgpu static_plugin`) | **Green in CI** (run `33030694040`) after the `array-bounds` fix — see [GCC](#gcc-and-warnings-as-errors) | | Minimal build | Not yet exercised — see [Open items](#open-items) | | Emscripten / ORT Web | **Builds, links, and runs in a browser** on a real GPU; registration verified in the binary *and* at runtime — see [ORT Web](#emscripten-and-ort-web) | -| Latency | **Measured on both plugin shapes** — ~4.8 us/node (wasm), ~1.2 us/node (native); the accepted tolerance is still unquantified, see [Open items](#open-items) | +| Latency | **Measured and root-caused** — ~4.8 us/node (wasm), ~1.2 us/node (native); the native cost is redundant per-kernel tensor-wrapper construction in the EP API adapter, not boundary-crossing cost, so it looks reducible — see [Open items](#open-items) | | PR | Draft [#32395](https://github.com/microsoft/onnxruntime/pull/32395); self-review feedback addressed — see [PR review feedback](#pr-review-feedback) | ## Branch @@ -668,6 +668,32 @@ native one charges less. That is consistent with the wasm cost being dominated b `static_plugin` wasm's per-kernel C++ path rather than by the plugin indirection itself, which is what the three-way `mismatch` run above already suggested. +### Root cause of the native per-node cost + +1.2 us/node is ~4,400 cycles, which is far more than a cross-DLL indirect call can account for, so both arms were +CPU-profiled to find out what it is actually spent on. The write-up is in +[native_plugin_latency_measurements.md](native_plugin_latency_measurements.md#where-the-per-node-cost-goes-cpu-profile). +Outcome: the overhead is **not** boundary-crossing cost. It is redundant per-kernel work that the built-in path does +not do — `ep::adapter::CreateTensorFromApiValue` rebuilds an `onnxruntime::Tensor` (nine C API round trips, an +allocator-name `std::string`, a `TensorShape` allocation) for every input and output of every node on every `Run`, +because the adapter's `OpKernelContext` is constructed fresh per `Compute()` and caches nothing beyond it. The +adapter symbols are absent from the built-in profile entirely, `Tensor`/`TensorShape` work doubles, and the total +extra CPU (+0.53 ms/rep) accounts for the entire extra wall latency (+0.42 ms/rep). + +Three method notes, since each cost time: + +- **ORT's own `-p` profiler cannot do this A/B.** The built-in arm emits 15,300 `Api` category events and the plugin + arm emits zero, so under `-p` the built-in arm measures *slower* and the result inverts. That asymmetry is also a + finding in its own right: the plugin path silently loses the WebGPU EP's `Api` profiling events, an observability + regression worth fixing independently of latency. +- **ETW/xperf symbolization does not work on this machine.** Traces collect fine but every frame renders as + `***unknown***` for every module, with symcache generated and with or without the Microsoft symbol server. Do not + spend time on it. +- **What worked was a purpose-built sampling profiler** (`D:\test\prof\sampler.cpp`, outside the repo): sample every + thread's instruction pointer at ~1 kHz, symbolize with dbghelp against local PDBs after exit. Needs no elevation. + The binaries were relinked with `/DEBUG` only — `LINK_FLAGS` already set `/OPT:REF,ICF,LBR` explicitly, so no + codegen changed and the A/B ratio was re-verified afterwards. + ## PR review feedback Draft PR [#32395](https://github.com/microsoft/onnxruntime/pull/32395) collected eleven inline self-review @@ -843,15 +869,20 @@ RTX 5060 Ti machine, `onnxruntime_test_all --gtest_filter=InferenceSessionTests. dropped; **decision: deferred until this branch lands**, at which point the link becomes valid and the whole set can go across together. -8. **Decide the latency tolerance.** The native half of the latency criterion is now measured — see - [Native shared-library plugin measurements](#native-shared-library-plugin-measurements). Both plugin shapes - cost per kernel node rather than a fixed amount, so no single percentage characterizes the overhead: it is - ~4-6% on real models and grows with node count. The criterion says latency must stay "within an accepted - tolerance", but that tolerance has never been quantified, and these results show the answer is not "zero". - Two gaps remain in the native numbers: session-creation cost was not compared (the plugin arm must also - discover and load the DLL, so it is likely worse in relative terms), and both arms ran with default WebGPU - options — graph capture in particular targets exactly the per-dispatch cost identified, and is worth a - follow-up. +8. **Decide the latency tolerance — but reduce the cost first.** The native half of the latency criterion is now + measured and root-caused — see + [Native shared-library plugin measurements](#native-shared-library-plugin-measurements). Profiling showed the + overhead is redundant per-kernel tensor-wrapper construction in the EP API adapter, not intrinsic + boundary-crossing cost, so it looks reducible: caching the `Tensor` wrapper across `Compute()` calls, and + avoiding the allocator-name `std::string` and the `TensorShape` allocation on the per-node path, are the obvious + candidates. Negotiating a tolerance around a fixable defect would be the wrong order. Two gaps also remain in the + native numbers: session-creation cost was not compared (the plugin arm must also discover and load the DLL, so it + is likely worse in relative terms), and both arms ran with default WebGPU options — graph capture in particular + targets exactly the per-dispatch cost identified, and is worth a follow-up. + +9. **Restore `Api` profiling events on the plugin path.** The built-in WebGPU EP emits 15,300 `Api` category events + under `-p`; the plugin path emits none. This is an observability regression independent of latency, and it makes + ORT's own profiler unusable for comparing the two paths. ## Environment notes From 3a01ce0cd63fb34236d3edb5c296b41ac9e2df38 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Tue, 8 Sep 2026 17:40:43 -0700 Subject: [PATCH 060/107] Record that the adapter-overhead hypothesis was tested and did not hold The CPU profile section concluded that the native plugin EP's per-node overhead is redundant per-kernel adapter work. That hypothesis was tested by memoizing the per-tensor OrtMemoryInfo reconstruction in the adapter OpKernelContext, which verifiably removed 57-78% of that work in the profile but produced no measurable latency change: the like-for-like dedicated bench_dispatch A/B is 1.0862 (baseline) against 1.0860 (with the cache). Converting sampler counts to critical-thread time shows why. The entire adapter plus C API layer costs ~59 us/rep more than built-in against a ~455 us/rep gap, i.e. ~13% - the same share the existing attribution table already reported. The data was never wrong; the narrative overstated it. Soften the wording in the Mechanism and tolerance sections accordingly. Also record that a GPU driver update moved the built-in control itself, so future A/B runs must re-measure their own baseline rather than comparing against the numbers committed here. The experimental change was reverted; it adds a cache and a non-obvious lifetime rule for no demonstrated benefit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../native_plugin_latency_measurements.md | 134 ++++++++++++++++-- 1 file changed, 124 insertions(+), 10 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md index b396176f609df..8446fcec316d8 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md @@ -324,8 +324,9 @@ Reproduce with `run_native_ab.ps1` (parses `Min Latency`, ignores the exit code) The 1.2 µs/node figure above is far too large to be call-transport cost — a cross-DLL indirect call through the C API is tens of cycles, while 1.2 µs at 3.696 GHz is ≈4,400 cycles. That gap motivated profiling the two arms to find out -what the cycles are actually spent on. The answer is that they are spent on **per-kernel marshalling work that the -built-in path does not do at all**, not on crossing the boundary. +what the cycles are actually spent on. The answer is that they are spent **host-side, not on crossing the boundary**. +Part of that host-side work is per-kernel marshalling that the built-in path does not do at all — but only a minority +part; see [Follow-up: testing the adapter hypothesis](#follow-up-testing-the-adapter-hypothesis). ### Method @@ -419,9 +420,15 @@ fresh per `Compute()` call and `resize()`s an input and an output `std::vector **Read [Follow-up: testing the adapter hypothesis](#follow-up-testing-the-adapter-hypothesis) before acting on +> this section.** An attempt to remove part of this adapter work produced **no measurable latency improvement**. The +> attribution table above is not contradicted by that result — it already assigns the adapter only 13.4% of the +> delta — but the wording of this section originally implied the adapter was the dominant cost, which the table does +> not support. ### Caveats @@ -436,8 +443,115 @@ the tens of cycles an indirect cross-module call would cost. ### Implication for the tolerance question -Open item 8 in the work log asks what tolerance is acceptable. These results argue against negotiating one yet: the -dominant identified cost is redundant work in the adapter, not an intrinsic property of the plugin boundary, so it -looks reducible — for example by caching the `Tensor` wrapper across `Compute()` calls, or by avoiding the -allocator-name string and shape allocation on the per-node path. The tolerance question is better asked after that -is attempted. +Open item 8 in the work log asks what tolerance is acceptable. The first reading of these results was that the +dominant cost is redundant adapter work and therefore looks reducible, so the tolerance question should wait. + +**That reading did not survive being tested.** The adapter is a minority cost by this profile's own attribution +(13.4% of the delta, or 28.7% if the `Tensor`/`TensorShape` row is counted as adapter-caused), and an attempt to +remove part of it changed nothing measurable. See the next section. + + +## Follow-up: testing the adapter hypothesis + +The section above concluded that the per-node overhead is redundant adapter work. That hypothesis was then tested +directly by removing a measurable part of that work and re-running the same A/B. **It produced no measurable latency +improvement.** This section records the experiment, because the negative result is what stops the next person from +repeating it. + +### The change that was tried + +`CreateTensorFromApiValue` rebuilds an `OrtMemoryInfo` on every tensor access, costing six C API round trips +(`GetAllocatorName`, `GetAllocatorType`, `GetDeviceType`, `GetMemoryType`, `GetVendorId`, `GetDeviceId`) plus a +`std::string` copy. Every input and output of a node normally lives on the same device, so almost all of these +reconstructions are identical. + +The change memoized them in the adapter `OpKernelContext`, keyed on the `const OrtMemoryInfo*` returned by +`GetTensorMemoryInfo`, and also removed a duplicated `GetMemoryType()` call. The cache must be scoped to a single +`Compute()` call: `OrtApis::GetTensorMemoryInfo` returns `&tensor.Location()`, a pointer *into the tensor*, so the +pointer identifies an allocator only while that tensor is alive. A session-lifetime cache would silently return +wrong memory info once an address was recycled. + +### The change worked, mechanically + +Verified with the same sampling profiler, 30,000 reps of `bench_dispatch`, shares normalized against total samples +(the sampler's tick rate differs between runs, so raw per-rep counts are not comparable across runs): + +| Quantity | before | after | change | +| --- | ---: | ---: | ---: | +| `OrtApis::GetTensorMemoryInfo` | — | — | −78% | +| `OrtApis::MemoryInfoGetDeviceType` | — | — | −73% | +| memory-info line, share of `CreateTensorFromApiValue` | 43.3% | 29.4% | −57% of total | +| `CreateTensorFromApiValue` self time | 8.93e-5 | 5.70e-5 | −36% | +| `KernelImpl::ComputeImpl` self time (control) | 8.53e-5 | 8.91e-5 | unchanged | + +### It made no measurable difference to latency + +All `bench_dispatch` runs, same session, same machine state, built-in arm untouched as a control: + +| run | harness | built-in | plugin | delta | ratio | +| --- | --- | ---: | ---: | ---: | ---: | +| baseline | dedicated, 12 rounds | 5.1656 | 5.6205 | +0.4549 | 1.0881 | +| baseline | dedicated, 16 rounds | 5.1253 | 5.5671 | +0.4418 | **1.0862** | +| baseline | 4-model, 8 rounds | 5.1912 | 5.6805 | +0.4893 | 1.094 | +| with cache | dedicated, 12 rounds | 5.1147 | 5.5543 | +0.4397 | **1.0860** | +| with cache | 4-model, 8 rounds | 5.1710 | 5.5353 | +0.3643 | 1.070 | + +The like-for-like pair — the two dedicated runs, same day, same harness — is **1.0862 against 1.0860, a dead heat**. +The one encouraging number (1.070) came from the 4-model harness, whose own baseline was simultaneously its worst +reading (1.094); the spread of that row across runs (delta 0.364–0.489 ms) is larger than the effect being looked for. + +Across all four models, with a same-session baseline: + +| model | baseline delta / ratio | with cache delta / ratio | +| --- | --- | --- | +| `bench_compute` | +0.0515 / 1.029 | +0.0439 / 1.027 | +| `bench_dispatch` | +0.4893 / 1.094 | +0.3643 / 1.070 | +| Qwen decode | +1.5917 / 1.050 | +2.1669 / 1.068 | +| Qwen prefill | +0.3782 / 1.009 | +0.5557 / 1.013 | + +Two rows improved and two got worse. Both Qwen rows are noise-dominated, as established earlier in this document, so +the mixed signs are expected and carry no information. + +Note that the earlier committed table cannot be used as the baseline here: a GPU driver update moved the built-in +control itself (Qwen decode 34.95 → 31.79 ms, prefill 47.66 → 43.03 ms). **Any future A/B on this branch must +re-measure its own baseline in the same session rather than comparing against numbers recorded in this document.** + +### Why: the addressable budget is small + +Converting sampler counts to time on the critical thread explains the null result. One sample is one tick of one +thread, so a symbol's time per rep is `samples / sampled_wall_ms * rep_ms`. The sampler samples *every* thread each +tick — around 25 threads here, most of them parked in wait stubs — so normalizing by total samples or by rep count +rather than by sampled wall time overstates or understates a symbol by the thread count. Adapter and C API code runs +on the single inference thread, so for those symbols this conversion is directly comparable to latency. + +| Scope | `ep::adapter::*` | `OrtApis::*` + `Ort::GetApi` | sum | +| --- | ---: | ---: | ---: | +| built-in (control) | 0.0 | 9.2 | 9.2 µs/rep | +| plugin, before the change | 26.7 | 41.2 | 67.9 µs/rep | +| plugin, after the change | 29.3 | 34.9 | 64.1 µs/rep | + +So the entire adapter plus C API layer costs about **59 µs/rep more than built-in, against a total gap of ~455 +µs/rep — roughly 13%.** That is the same 13.4% the attribution table in the previous section already reported. Even +perfect elimination of all adapter and C API cost cannot close more than about one eighth of the deficit, and the +change actually made addressed only part of that eighth. An effect of that size is below this setup's noise floor, +so the experiment does not prove the change is worthless — it bounds it at under roughly 5% of the gap. + +The change was reverted rather than kept: it adds a cache class and a non-obvious lifetime rule for no demonstrated +benefit. + +### What this leaves + +The adapter is a real, new, and reducible cost, but it is a minority one. The remaining ~85% of the gap is +elsewhere. The plugin-minus-built-in profile points at `dawn` refcounting, `std::string`, `TensorShape`, and +`ComputeContext::Input` all being heavier in the plugin build — a pattern consistent with **lost cross-module +inlining** once the WebGPU code moves into its own DLL, which no amount of adapter tuning can recover. Candidates +worth testing before revisiting the adapter: + +- link-time code generation / whole-program optimization on the plugin DLL, to test the lost-inlining theory; +- graph capture (`enableGraphCapture`), which targets per-dispatch cost directly and has never been measured here; +- confirming whether the GPU-driver share of the delta (`nvwgf2umx.dll` +0.085 samples/rep) is genuine extra work or + a wall-time-proportional polling artifact — i.e. consequence rather than cause. + +A caveat that applies to all of the above: this is still a flat, leaf-address profile. It measures self time and +cannot prove that a `malloc` or `dawn` sample was reached *from* any particular caller. More samples improve +precision, not attribution. Settling that would need stack walking, which has not been implemented. \ No newline at end of file From 475f2cb997a7b481847558173599581204e1f2a4 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Wed, 9 Sep 2026 11:20:39 -0700 Subject: [PATCH 061/107] Update WASM size comparison with both shipped WebGPU artifacts The npm package ships two WebGPU WASM builds, asyncify and JSPI, but only asyncify had been measured. Rebuilt all four combinations (static_lib vs static_plugin, asyncify vs JSPI) from scratch using the release pipeline flag set, and recorded raw, gzip and brotli sizes. All deltas are under 1%. The asyncify raw number reproduces (-61,968 B originally, -62,026 B now), but the raw sign is not stable across the two artifacts and compressed size grows slightly in both, so the previous blanket "no size regression" claim is re-scoped to what was measured. --- .../static_plugin_ep_registration_design.md | 64 ++++++++++++++----- 1 file changed, 48 insertions(+), 16 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md index 9a9c7642a4158..d3918deb01917 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -570,24 +570,56 @@ That also means the only existing lever that trims plugin-EP machinery is `--min #### Measured size comparison -Both modes were then built locally with the exact CI flag set (`Release`, SIMD + threads, asyncify, WebNN on, -`--disable_rtti`, `--enable_wasm_api_exception_catching`, and the full reduced-size set above) and the artifacts -compared: +The ORT Web npm package ships **two** WebGPU WASM artifacts, not one, and both were measured. The release path +(`npm-packaging-pipeline.yml` → `templates/web-ci.yml` → `templates/linux-wasm-ci.yml`) and the GitHub Actions path +(`web.yml` → `linux-wasm-ci-build-and-test-workflow.yml`) pass an identical flag set for the WebGPU legs, which +differ from each other only in their exception/stack-switching mode: + +| Leg | Artifact | Distinguishing flag | +| --- | --- | --- | +| `wasm_inferencing_webgpu` | `ort-wasm-simd-threaded.asyncify.{wasm,mjs}` | `--enable_wasm_api_exception_catching` | +| `wasm_inferencing_webgpu_jspi` | `ort-wasm-simd-threaded.jspi.{wasm,mjs}` | `--enable_wasm_jspi` | + +Four clean from-scratch builds were run (both EP modes × both legs) with the exact release flag set: `Release`, +SIMD + threads, WebNN on, `--disable_rtti`, `--target onnxruntime_webassembly`, and the full reduced-size set +above. Compressed sizes are included because that is what is actually delivered over the wire; they were produced +with gzip level 9 and brotli quality 11. + +`ort-wasm-simd-threaded.asyncify.wasm`: + +| Metric | `static_lib` | `static_plugin` | Delta | +| --- | ---: | ---: | ---: | +| raw | 26,549,423 B | 26,487,397 B | **−62,026 B (−0.234%)** | +| gzip | 6,537,701 B | 6,566,088 B | **+28,387 B (+0.434%)** | +| brotli | 3,953,625 B | 3,971,550 B | **+17,925 B (+0.453%)** | -| Artifact | `static_lib` | `static_plugin` | Delta | +`ort-wasm-simd-threaded.jspi.wasm`: + +| Metric | `static_lib` | `static_plugin` | Delta | | --- | ---: | ---: | ---: | -| `ort-wasm-simd-threaded.asyncify.wasm` | 26,550,879 B | 26,488,911 B | **−61,968 B (−0.23%)** | -| `ort-wasm-simd-threaded.asyncify.mjs` | 53,249 B | 53,249 B | 0 | - -(Both binaries rebuilt after merging `main`. An earlier pre-merge pair measured −52,978 B / −0.20%; the merge added -kernels to both builds and did not change the relationship.) - -**There is no size regression** — `static_plugin` is marginally smaller. Both configurations compiled cleanly with -every reduced-size flag, which empirically confirms the analysis above. The plugin build plausibly comes out ahead -because it drops the internal `IExecutionProvider` / `KernelRegistry` glue and the built-in `WebGpuEpFactory` in -`ep_library_internal.cc` in favour of the header-only adapters, but that attribution is not separately measured. -For reference, the same `static_plugin` configuration without any reduced-size flags is 40.9 MB, so those flags are -worth ~15 MB and remain essential regardless of EP mode. +| raw | 16,627,577 B | 16,687,113 B | **+59,536 B (+0.358%)** | +| gzip | 4,099,914 B | 4,126,147 B | **+26,233 B (+0.640%)** | +| brotli | 2,651,643 B | 2,674,520 B | **+22,877 B (+0.863%)** | + +The accompanying `.mjs` files are byte-identical between the two EP modes in both legs (53,300 B asyncify, 51,204 B +JSPI); their compressed sizes differ by at most 23 B. + +**Every delta is under 1%, which is the substantive conclusion.** Two details are worth recording so the numbers +are not over-read: + +- *The raw sign is not stable across the two artifacts.* `static_plugin` is marginally smaller raw on asyncify but + marginally larger raw on JSPI. An earlier revision of this document reported only the asyncify raw number and + generalised it to "no size regression"; that generalisation was too broad. The asyncify measurement itself + reproduces well — it was −61,968 B when first taken and −62,026 B in this from-scratch rerun at a later commit. + +- *Compressed size grows slightly in both legs*, by +0.4% to +0.9%, so the direction is consistent once the + artifacts are compressed. Why compressed size can rise while raw size falls was not investigated; no + symbol-level or per-section attribution was done, and none of the earlier speculation about *which* code the + plugin build drops or adds has been measured. Given the magnitude, this was not pursued further. + +Both configurations compiled cleanly with every reduced-size flag in both legs, which empirically confirms the +compatibility analysis above. For reference, the same `static_plugin` asyncify configuration without any +reduced-size flags is 40.9 MB, so those flags are worth ~15 MB and remain essential regardless of EP mode. #### `--enable_wasm_api_exception_catching` and the EP API adapters From 5a84e15bae203b1af21cb52ba6cd03800b38d00e Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Wed, 9 Sep 2026 14:10:56 -0700 Subject: [PATCH 062/107] Restrict Environment::CreateAndRegisterStaticPluginEps to OrtEnv via a passkey The function was public but only safe to call from OrtEnv::GetOrCreateInstance: it must run after the OrtEnv singleton is published, while the OrtEnv creation mutex is held, and it deliberately does not take Environment::mutex_. Those requirements were documented only in a comment, so any caller could get them wrong. Its sibling CreateAndRegisterInternalEps is already private. Add a StaticPluginEpRegistrationToken passkey that only OrtEnv can construct, making the restriction compile-enforced and self-documenting at the call site. Also record the locking requirement in the function comment, which previously mentioned only the ordering one. The token deletes copy and move in addition to keeping its default constructor private. An implicitly declared copy constructor is public, which would let a non-friend manufacture a token from arbitrary storage. The only call site passes a prvalue, so guaranteed copy elision means the deletions cost nothing. The passkey also depends on C++20: before C++20 a class whose only constructor is defaulted on its first declaration is still an aggregate, so a non-friend could bypass the private constructor via brace initialization. ORT requires C++20. Verified by compiling both bypasses, which are rejected with C2248 and C2280 respectively, while the call from OrtEnv compiles. --- .../onnxruntime/core/session/environment.h | 19 ++++++++++++++++--- onnxruntime/core/session/environment.cc | 2 +- onnxruntime/core/session/ort_env.cc | 6 ++++-- 3 files changed, 21 insertions(+), 6 deletions(-) diff --git a/include/onnxruntime/core/session/environment.h b/include/onnxruntime/core/session/environment.h index 9f61f19e0a601..6ee9eee03a2f8 100644 --- a/include/onnxruntime/core/session/environment.h +++ b/include/onnxruntime/core/session/environment.h @@ -140,15 +140,28 @@ class Environment { Status RegisterExecutionProviderLibrary(const std::string& registration_name, const ORTCHAR_T* lib_path); Status UnregisterExecutionProviderLibrary(const std::string& registration_name); + /** + * Passkey that restricts CreateAndRegisterStaticPluginEps() to OrtEnv, which is the only caller able to satisfy + * that function's ordering and locking requirements. Only OrtEnv can construct one. + */ + class StaticPluginEpRegistrationToken { + private: + StaticPluginEpRegistrationToken() = default; + ORT_DISALLOW_COPY_ASSIGNMENT_AND_MOVE(StaticPluginEpRegistrationToken); + friend struct ::OrtEnv; + }; + /** * Register the plugin execution providers that are statically linked into the ORT binary. * * This must be called after the OrtEnv singleton that owns this Environment has been published, because a * statically linked plugin EP uses the public ORT API, and any OrtEnv API it calls must be able to find the - * instance. See docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/ - * static_plugin_ep_registration_design.md. + * instance. It must also be called while the OrtEnv creation mutex is held and before any other thread can reach + * this Environment, because it intentionally does not take mutex_. + * + * The passkey parameter limits the set of possible callers to OrtEnv. */ - Status CreateAndRegisterStaticPluginEps(); + Status CreateAndRegisterStaticPluginEps(StaticPluginEpRegistrationToken); // convert an OrtEpFactory* to EpFactoryInternal* if possible. EpFactoryInternal* GetEpFactoryInternal(OrtEpFactory* factory) const { diff --git a/onnxruntime/core/session/environment.cc b/onnxruntime/core/session/environment.cc index 453b0430a1b63..ee6d6422b5758 100644 --- a/onnxruntime/core/session/environment.cc +++ b/onnxruntime/core/session/environment.cc @@ -644,7 +644,7 @@ Status Environment::CreateAndRegisterInternalEps() { return Status::OK(); } -Status Environment::CreateAndRegisterStaticPluginEps() { +Status Environment::CreateAndRegisterStaticPluginEps(StaticPluginEpRegistrationToken) { // Intentionally does not take mutex_. That is safe because the caller holds the OrtEnv creation mutex, so no other // thread can obtain a reference to this Environment yet. Taking mutex_ here would deadlock if a plugin EP called an // Environment API from its OrtEpFactory::GetSupportedDevices implementation. diff --git a/onnxruntime/core/session/ort_env.cc b/onnxruntime/core/session/ort_env.cc index 49ad00c3a233f..1ba5361ee82ed 100644 --- a/onnxruntime/core/session/ort_env.cc +++ b/onnxruntime/core/session/ort_env.cc @@ -100,9 +100,11 @@ OrtEnvPtr OrtEnv::GetOrCreateInstance(const OrtEnv::LoggingManagerConstructionIn #if !defined(ORT_MINIMAL_BUILD) // Register statically linked plugin EPs *after* p_instance_ is published. They use the public ORT API, so any // OrtEnv API they call while enumerating their devices must be able to find the instance. m_ is recursive so - // that such a call from this thread doesn't self-deadlock. See + // that such a call from this thread doesn't self-deadlock. The passkey argument documents that OrtEnv is the + // only caller able to satisfy those requirements. See // docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md - status = p_instance_->GetEnvironment().CreateAndRegisterStaticPluginEps(); + status = p_instance_->GetEnvironment().CreateAndRegisterStaticPluginEps( + Environment::StaticPluginEpRegistrationToken{}); if (!status.IsOK()) { return OrtEnvPtr(nullptr, OrtEnv::Release); From c32806d907003b7e12862f170b222a0e92d16f38 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Wed, 9 Sep 2026 15:00:01 -0700 Subject: [PATCH 063/107] Correct the stale self-deadlock comment on CreateAndRegisterInternalEps The comment said the internal WebGPU EP factory cannot query the OrtEnv singleton from GetSupportedDevices because OrtEnv's creation mutex is already held on this thread and the query would self-deadlock. That was true when it was written, but making OrtEnv::m_ a recursive_mutex to support statically linked plugin EPs removed the deadlock, leaving the stated reason wrong. Environment::mutex_ is not held there either, since neither Environment::Create nor Environment::Initialize takes it. The conclusion still holds, for a different reason: internal EPs are registered from Environment::Create, which runs before OrtEnv publishes p_instance_, so there is no instance for the query to find. Statically linked plugin EPs can query it precisely because they are registered after publication, which is why the two registration paths are separate. --- onnxruntime/core/session/environment.cc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/onnxruntime/core/session/environment.cc b/onnxruntime/core/session/environment.cc index ee6d6422b5758..a47af1875c14f 100644 --- a/onnxruntime/core/session/environment.cc +++ b/onnxruntime/core/session/environment.cc @@ -629,8 +629,8 @@ Status Environment::RegisterExecutionProviderLibrary(const std::string& registra Status Environment::CreateAndRegisterInternalEps() { // Capture allow_virtual_devices here (lock-free) and pass it to the internal EP factories at // construction. The internal WebGPU EP factory needs it in GetSupportedDevices but cannot query the - // OrtEnv singleton there: internal EPs are registered while OrtEnv's creation mutex is already held on - // this thread, so the query would self-deadlock. + // OrtEnv singleton there: internal EPs are registered from Environment::Create, before OrtEnv + // publishes p_instance_, so there is no instance to find. const bool allow_virtual_devices = num_allow_virtual_device_uses_ > 0; auto internal_ep_libraries = EpLibraryInternal::CreateInternalEps(allow_virtual_devices); for (auto& ep_library : internal_ep_libraries) { From c7083457588fc1f692b26c44a132ca16963c7be3 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Wed, 9 Sep 2026 15:31:04 -0700 Subject: [PATCH 064/107] Address PR review comments on code comments - cmake/onnxruntime_providers_webgpu.cmake: drop the entry-point prefix explanation. It duplicates the more complete one at onnxruntime/core/providers/webgpu/ep/api.cc:15-20, which also covers why the shared library build stays unprefixed. - environment.h: drop the mutex_ sentence from the CreateAndRegisterStaticPluginEps doc comment. The implementation in environment.cc already carries that rationale, which is where it belongs. - ort_env.cc: make the static plugin EP registration comment self-contained instead of pointing at the design doc. - matmul_nbits_mlp.cc: shorten the MakeString comment to the one fact a future editor needs. --- cmake/onnxruntime_providers_webgpu.cmake | 3 --- include/onnxruntime/core/session/environment.h | 3 +-- .../contrib_ops/webgpu/quantization/matmul_nbits_mlp.cc | 4 +--- onnxruntime/core/session/ort_env.cc | 4 +--- 4 files changed, 3 insertions(+), 11 deletions(-) diff --git a/cmake/onnxruntime_providers_webgpu.cmake b/cmake/onnxruntime_providers_webgpu.cmake index ab29c7a01f88c..e389a81e0695a 100644 --- a/cmake/onnxruntime_providers_webgpu.cmake +++ b/cmake/onnxruntime_providers_webgpu.cmake @@ -48,9 +48,6 @@ onnxruntime_add_include_to_target(onnxruntime_providers_webgpu onnxruntime_common onnx onnx_proto flatbuffers::flatbuffers Boost::mp11 safeint_interface) - # Prefix the plugin EP entry points so that multiple statically linked plugin EPs can coexist in one binary. - # ORT core declares the prefixed names in onnxruntime/core/session/plugin_ep/ep_static_plugins.cc. - # # ORT_PLUGIN_EP_SKIP_API_MANUAL_INIT tells the shared plugin EP headers that OrtGetApiBase() is available # in-process, so the C++ API must not be built with ORT_API_MANUAL_INIT (the rest of the binary is not). target_compile_definitions(onnxruntime_providers_webgpu PRIVATE diff --git a/include/onnxruntime/core/session/environment.h b/include/onnxruntime/core/session/environment.h index 6ee9eee03a2f8..a8bb00fc1ef3d 100644 --- a/include/onnxruntime/core/session/environment.h +++ b/include/onnxruntime/core/session/environment.h @@ -156,8 +156,7 @@ class Environment { * * This must be called after the OrtEnv singleton that owns this Environment has been published, because a * statically linked plugin EP uses the public ORT API, and any OrtEnv API it calls must be able to find the - * instance. It must also be called while the OrtEnv creation mutex is held and before any other thread can reach - * this Environment, because it intentionally does not take mutex_. + * instance. * * The passkey parameter limits the set of possible callers to OrtEnv. */ diff --git a/onnxruntime/contrib_ops/webgpu/quantization/matmul_nbits_mlp.cc b/onnxruntime/contrib_ops/webgpu/quantization/matmul_nbits_mlp.cc index 1693ab4ea2028..53b42870d0b2a 100644 --- a/onnxruntime/contrib_ops/webgpu/quantization/matmul_nbits_mlp.cc +++ b/onnxruntime/contrib_ops/webgpu/quantization/matmul_nbits_mlp.cc @@ -45,9 +45,7 @@ std::string EmitGateActivationExpr(MlpActivationKind kind, std::string_view gate switch (kind) { case MlpActivationKind::Silu: // SiLU(x) = x * sigmoid(x) - // Built with MakeString rather than a std::string concatenation chain: GCC 14 at -O3 reports a false - // positive -Warray-bounds through the inlined basic_string move constructor when this translation unit - // is compiled with the EP adapter headers in the precompiled header. + // MakeString avoids a GCC 14 -Warray-bounds false positive from std::string concatenation here. return MakeString(gate_var, " * (one / (one + exp(-", gate_var, ")))"); } ORT_THROW("MatMulNBitsMlp: unhandled MlpActivationKind ", static_cast(kind)); diff --git a/onnxruntime/core/session/ort_env.cc b/onnxruntime/core/session/ort_env.cc index 1ba5361ee82ed..7786a051d6861 100644 --- a/onnxruntime/core/session/ort_env.cc +++ b/onnxruntime/core/session/ort_env.cc @@ -100,9 +100,7 @@ OrtEnvPtr OrtEnv::GetOrCreateInstance(const OrtEnv::LoggingManagerConstructionIn #if !defined(ORT_MINIMAL_BUILD) // Register statically linked plugin EPs *after* p_instance_ is published. They use the public ORT API, so any // OrtEnv API they call while enumerating their devices must be able to find the instance. m_ is recursive so - // that such a call from this thread doesn't self-deadlock. The passkey argument documents that OrtEnv is the - // only caller able to satisfy those requirements. See - // docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md + // that such a call from this thread doesn't self-deadlock. status = p_instance_->GetEnvironment().CreateAndRegisterStaticPluginEps( Environment::StaticPluginEpRegistrationToken{}); From e2915c69073b777161d168e31f687e544b07f8be Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Wed, 9 Sep 2026 16:05:49 -0700 Subject: [PATCH 065/107] Select the WebGPU EP by OrtEpDevice without claiming it is a plugin EP Selection by OrtEpDevice is not specific to the plugin EP. Both flavours register an OrtEpDevice under the canonical name kWebGpuExecutionProvider: the built-in EP via EpLibraryInternal::CreateWebGpuEp when ORT_USE_EP_API_ADAPTERS is off, and the static plugin EP via CreateStaticPluginEpLibraries when it is on. So the bundle works against either wasm flavour, which a runtime control had already shown. Rename selectAsPluginEp to selectByEpDevice and drop the claim that the paired wasm binary is an ORT_USE_EP_API_ADAPTERS build. That is false for every wasm ORT Web ships today: both WebGPU wasm builds pass a bare --use_webgpu, which is static_lib. Also correct the same stale self-deadlock comment in ep_factory_webgpu.h that was already fixed in environment.cc. OrtEnv::m_ is recursive, so a query from this thread would not deadlock; the actual reason the factory cannot query the singleton is that p_instance_ is not published yet. --- js/web/lib/wasm/session-options.ts | 18 +++++++++--------- .../core/session/plugin_ep/ep_factory_webgpu.h | 5 ++--- 2 files changed, 11 insertions(+), 12 deletions(-) diff --git a/js/web/lib/wasm/session-options.ts b/js/web/lib/wasm/session-options.ts index 85db499585efa..f17d49e7d48e5 100644 --- a/js/web/lib/wasm/session-options.ts +++ b/js/web/lib/wasm/session-options.ts @@ -117,10 +117,8 @@ const setExecutionProviders = async ( for (const ep of executionProviders) { let epName = typeof ep === 'string' ? ep : ep.name; const epOptions: Array<[number, number]> = []; - // True when the EP is selected as a plugin EP (by OrtEpDevice) rather than by ORT's built-in EP name table. - // The built-in name table has no WebGPU entry in an ORT_USE_EP_API_ADAPTERS build, which is what the wasm - // binary paired with this bundle is built as (see BUILD_DEFS.DISABLE_WEBGPU below). - let selectAsPluginEp = false; + // True when the EP is selected by OrtEpDevice rather than by ORT's built-in EP name table. + let selectByEpDevice = false; // check EP name switch (epName) { @@ -141,10 +139,11 @@ const setExecutionProviders = async ( break; case 'webgpu': if (!BUILD_DEFS.DISABLE_WEBGPU) { - // The plugin EP is registered under its canonical name (OrtEpFactory::GetName), not the short 'WebGPU' - // alias that ORT's built-in EP name table used. + // Select by OrtEpDevice, under the EP's canonical name (OrtEpFactory::GetName) rather than the short + // 'WebGPU' alias of the built-in EP name table. This is independent of how the paired wasm binary + // builds the WebGPU EP: the built-in EP and the plugin EP both register an OrtEpDevice under this name. epName = 'WebGpuExecutionProvider'; - selectAsPluginEp = true; + selectByEpDevice = true; let customDevice: GPUDevice | undefined; if (typeof ep !== 'string') { @@ -243,7 +242,7 @@ const setExecutionProviders = async ( } let appendErrorCode: number; - if (selectAsPluginEp) { + if (selectByEpDevice) { const epDevicesByEpName = getEpDevicesByEpName(); const epDevices = epDevicesByEpName.get(epName); if (!epDevices) { @@ -254,7 +253,8 @@ const setExecutionProviders = async ( // Select a single OrtEpDevice, even though more than one may match the EP name. // // Necessary: the WebGPU EP backs a session with one Dawn device, so its factory rejects any device - // count other than 1 (see Factory::CreateEpImpl in core/providers/webgpu/ep/factory.cc). + // count other than 1 (see WebGpuEpFactory::CreateIExecutionProvider for the built-in EP and + // Factory::CreateEpImpl in core/providers/webgpu/ep/factory.cc for the plugin EP). // // Safe to take the first: the factory reads nothing from the device except whether it is virtual, and // Emscripten device discovery reports a single GPU entry derived from navigator.gpu, so there is at diff --git a/onnxruntime/core/session/plugin_ep/ep_factory_webgpu.h b/onnxruntime/core/session/plugin_ep/ep_factory_webgpu.h index dcb0b1cef6ee1..f8542b4db485a 100644 --- a/onnxruntime/core/session/plugin_ep/ep_factory_webgpu.h +++ b/onnxruntime/core/session/plugin_ep/ep_factory_webgpu.h @@ -13,9 +13,8 @@ namespace onnxruntime { class WebGpuEpFactory : public EpFactoryInternalImpl { public: // allow_virtual_devices is captured at construction (from the OrtEnv config "allow_virtual_devices") - // rather than queried from the OrtEnv singleton inside GetSupportedDevices: internal EPs are - // registered while the OrtEnv creation mutex is already held on this thread, so querying the singleton - // there would self-deadlock. + // rather than queried from the OrtEnv singleton inside GetSupportedDevices: internal EPs are registered + // from Environment::Create, before OrtEnv publishes p_instance_, so there is no instance to find. explicit WebGpuEpFactory(bool allow_virtual_devices) : EpFactoryInternalImpl(kWebGpuExecutionProvider, "Microsoft", OrtDevice::VendorIds::MICROSOFT), allow_virtual_devices_{allow_virtual_devices} { From 21abbc629d48aeecf7b61359db535ff5a4a18fe0 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 10 Sep 2026 14:21:23 -0700 Subject: [PATCH 066/107] Document OrtEpFactory::GetSupportedDevices constraints Tighten the EP API documentation for OrtEpFactory::GetSupportedDevices to state what an implementation may rely on: - Its own OrtEpDevice instances are not registered yet, so OrtApi::GetEpDevices returns an incomplete list. This is a property of EP library registration generally, not of static linking: EpInfo::Create calls GetSupportedDevices before RegisterExecutionProviderLibrary appends to execution_devices_. - Environment configuration entries are fully populated before any library is registered, so OrtEpApi::GetEnvConfigEntries may be called. - ORT holds internal environment locks across the call, so calling OrtApi::RegisterExecutionProviderLibrary or OrtApi::UnregisterExecutionProviderLibrary deadlocks on the non-recursive Environment::mutex_. Also replace the stale \return description, which claimed a bool return and referenced a nonexistent 'device' parameter, with the OrtStatus return value snippet used by the other ORT_API2_STATUS members. Clarify in ort_env.h that OrtEnv::m_ is recursive solely to support static plugin EP registration, and record the partial-visibility behavior in the static plugin EP registration design doc as an accepted side effect. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 473dd356-5e09-4a65-9bf6-a6b768f01dd9 --- .../static_plugin_ep_registration_design.md | 13 +++++++++++-- .../core/session/onnxruntime_ep_c_api.h | 15 ++++++++++++++- onnxruntime/core/session/ort_env.h | 6 ++++-- 3 files changed, 29 insertions(+), 5 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md index d3918deb01917..a0c40fd63246c 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -170,7 +170,9 @@ Properties: - The entire `OrtApi` behaves normally for provider code. `TryGetInstance` finds a published instance, and a re-entrant `CreateEnv` acquires the recursive lock and increments the reference count. -- No new contract for provider authors, and nothing for a future API author to remember. +- No new contract for provider authors, and nothing for a future API author to remember. The constraints that do + apply to `GetSupportedDevices` are the ones dynamic plugin EP registration already imposes, now documented on + `OrtEpFactory::GetSupportedDevices` in `onnxruntime_ep_c_api.h`. - No race. Other threads block on `m_` for the duration, which is already true across `Environment::Create`. - The reference count cannot reach zero mid-construction, because the creating thread's reference is taken first. - Provider sources need no conditional compilation for the two linkage modes. @@ -179,7 +181,14 @@ Costs and limits: - Failure during registration requires explicit teardown of the just-published instance. - A recursive mutex is normally undesirable. Here it states the actual invariant: the environment creation path can - legitimately re-enter the environment accessor on the same thread. + legitimately re-enter the environment accessor on the same thread. Static plugin EP registration is the only reason + `OrtEnv::m_` is recursive; it was a `std::mutex` before this change and no other code path requires recursion. +- A static plugin EP's `GetSupportedDevices` runs against an environment in which its own `OrtEpDevice` instances are + not yet registered, so `OrtApi::GetEpDevices` returns an incomplete list. This is not specific to static linking: + `EpInfo::Create` calls `GetSupportedDevices` before `RegisterExecutionProviderLibrary` appends to + `execution_devices_`, so a dynamically registered plugin EP sees the same thing. Registration also proceeds one + library at a time, so with more than one static plugin EP the Nth would observe the first N-1, making the visible + set depend on `CreateStaticPluginEpLibraries` ordering. Accepted as a side effect. - Callers of `onnxruntime::Environment::Create` do not get static plugin EPs. In the tree this is one production call site plus tests and orttraining sample binaries. - Provider code that starts a thread which calls `CreateEnv` and then joins it will deadlock. This is already true of diff --git a/include/onnxruntime/core/session/onnxruntime_ep_c_api.h b/include/onnxruntime/core/session/onnxruntime_ep_c_api.h index 0424a95c2d2c1..9537f8b98be7e 100644 --- a/include/onnxruntime/core/session/onnxruntime_ep_c_api.h +++ b/include/onnxruntime/core/session/onnxruntime_ep_c_api.h @@ -2795,7 +2795,20 @@ struct OrtEpFactory { * \param[in] max_ep_devices The maximum number of OrtEpDevices that can be added to ep_devices. * Current default is 8. This can be increased if needed. * \param[out] num_ep_devices The number of EP devices added to ep_devices. - * \return true if the factory can create an execution provider that uses `device`. + * + * \snippet{doc} snippets.dox OrtStatus Return Value + * + * \remarks Called during execution provider library registration. The OrtEpDevice instances returned are added to + * the environment after this returns, so an implementation must not depend on the environment's registered + * execution provider devices. OrtApi::GetEpDevices will not include this factory's devices, and whether it + * includes devices from other libraries depends on registration order. + * + * Environment configuration entries are populated before any library is registered, so + * OrtEpApi::GetEnvConfigEntries may be called from this function. + * + * ORT holds internal environment locks across this call. An implementation must not call + * OrtApi::RegisterExecutionProviderLibrary or OrtApi::UnregisterExecutionProviderLibrary, as that + * deadlocks, and should otherwise limit the ORT APIs it calls and the work it does. * * \since Version 1.22. */ diff --git a/onnxruntime/core/session/ort_env.h b/onnxruntime/core/session/ort_env.h index 47613ec99ebf3..fbfaa574c62f3 100644 --- a/onnxruntime/core/session/ort_env.h +++ b/onnxruntime/core/session/ort_env.h @@ -80,8 +80,10 @@ struct OrtEnv { // tracking active users. It is set to nullptr when the last reference is released // (and not shutting down). static OrtEnv* p_instance_; - // Recursive because statically linked plugin EPs are registered while this mutex is held, and a plugin EP may - // call an OrtEnv API (e.g. OrtEpApi::GetEnvConfigEntries) from OrtEpFactory::GetSupportedDevices. + // Recursive solely to support static plugin EP registration, which happens while this mutex is held: a plugin EP + // may call an OrtEnv API (e.g. OrtEpApi::GetEnvConfigEntries) from OrtEpFactory::GetSupportedDevices, which + // re-enters TryGetInstance(). No other code path requires recursive locking, so this can revert to std::mutex if + // static plugin EPs are no longer registered here. static std::recursive_mutex m_; static int ref_count_; From cadd1a5d9a89cd0e599aed9362142e7d92a2e191 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 10 Sep 2026 14:10:12 -0700 Subject: [PATCH 067/107] docs: record remaining WebGPU consumers and simplify the disposition table Add Android AAR, Java packages, and macOS native packages, from PR review feedback. Each ships WebGPU today and had no recorded disposition: the default full AAR build enables it; OrtProvider.WEBGPU and SessionOptions.addWebGPU() are public API, with the JAR carrying the Dawn and DXC native libraries; and macOS packaging builds with --use_webgpu, so the released onnxruntime-osx tarball has it in libonnxruntime.dylib and also supplies the native NuGet package. The Apple framework build does not enable WebGPU. Dispositions are recorded as not yet decided, to be filled in with the rest of the package and platform inventory. Drop the extraction requirement column. Six of its seven values amounted to Required, and the section already opens by saying every consumer needs a disposition before the built-in implementation is removed. The seventh was a task about the inventory rather than a requirement, so it goes too. Name the Python and NuGet plugin packaging in the provider-isolation summary so its ownership is visible. Say consolidated rather than moved, since the package sources already live under plugin-ep-webgpu and only the packaging pipelines are still outside it. State what happens to bundled Node WebGPU if source transfer lands first: the bundled build consumes the pinned external source until the replacement package ships, or transfer waits for it. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .../webgpu_ep_extraction.md | 23 +++++++++++-------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md b/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md index 09e7b66e6fca5..66fd707f25071 100644 --- a/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md +++ b/docs/design/webgpu_ep_extraction/webgpu_ep_extraction.md @@ -47,7 +47,7 @@ The effort is divided into four workstreams: | Identifier | Workstream | Primary outcome | | --- | --- | --- | | `plugin-boundary` | [Plugin boundary and Web/Wasm integration](plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md) | Static and dynamic builds use the same public plugin EP boundary, including the ORT Web browser bridge | -| `provider-isolation` | [Provider isolation and repository migration](provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md) | WebGPU-owned code, dependencies, tests, and existing plugin packaging move under `plugin-ep-webgpu/` and then to an independent repository | +| `provider-isolation` | [Provider isolation and repository migration](provider_isolation_and_repository_migration/provider_isolation_and_repository_migration_workstream.md) | WebGPU-owned code, dependencies, tests, and the Python and NuGet plugin packaging are consolidated under `plugin-ep-webgpu/` and then move to an independent repository | | `test-conformance` | [Test ownership and operator conformance](test_ownership_and_conformance/test_ownership_and_conformance_workstream.md) | Existing coverage is preserved, every test has an owner, and portable conformance coverage protects the external provider | | `node-migration` | [Node plugin migration](node_plugin_migration/node_plugin_migration_workstream.md) | Existing bundled Node WebGPU support is replaced by an explicitly consumable plugin without regressing current users | @@ -80,13 +80,16 @@ provider being extracted. This is handled in Every consumer that receives WebGPU today needs a recorded disposition before the built-in implementation is removed from ORT. Platform and architecture details remain to be inventoried in the individual workstreams. -| Consumer or package | Disposition | Extraction requirement | -| --- | --- | --- | -| `onnxruntime-web` | Consume a pinned external WebGPU source revision through static plugin registration | Required | -| Python WebGPU plugin package | Move the existing optional plugin package and release pipeline | Required | -| WebGPU NuGet plugin package | Move the existing optional plugin package and release pipeline | Required | -| Node WebGPU support | Provide a tested replacement for the WebGPU implementation currently bundled in `onnxruntime-node`; final package naming is open | Required before bundled support is removed | -| `onnxruntime-webgpu` on PyPI | Publication has already stopped; do not resurrect it or convert it into a plugin-dependent package | Confirm whether any other retired package needs the same treatment | +| Consumer or package | Disposition | +| --- | --- | +| `onnxruntime-web` | Consume a pinned external WebGPU source revision through static plugin registration | +| Python WebGPU plugin package | Move the existing plugin package and packaging pipeline | +| WebGPU NuGet plugin package | Move the existing plugin package and packaging pipeline | +| Node WebGPU support | Provide a tested replacement for the WebGPU implementation currently bundled in `onnxruntime-node`; final package naming is open | +| Android AAR | Not yet decided. The default full AAR build enables WebGPU | +| Java packages | Not yet decided. `OrtProvider.WEBGPU` and `SessionOptions.addWebGPU()` are public API, and the JAR carries the Dawn and DXC native libraries | +| macOS native packages | Not yet decided. macOS packaging builds with `--use_webgpu`, so the released `onnxruntime-osx-*` tarball has WebGPU in `libonnxruntime.dylib`, and that tarball also supplies the native NuGet package. The Apple `.framework` build does not enable WebGPU | +| Python `onnxruntime-webgpu` package | Publication has already stopped. Do not resurrect it or convert it into a plugin-dependent package | Hosts that do not ship WebGPU support today are outside this table. Adding one is a separate decision and is not an extraction prerequisite. @@ -124,7 +127,9 @@ The other convergence points between workstreams: - Test classification determines which tests move with the provider and which remain in ORT. Source transfer waits on it. - The Node workstream consumes generic plugin loading from ORT and native WebGPU artifacts from the external - provider. It does not gate source transfer, but bundled Node WebGPU cannot be removed until it lands. + provider. It does not gate source transfer, but bundled Node WebGPU cannot be removed until it lands. If transfer + comes first, the bundled Node build has to consume the pinned external source until the replacement package ships; + the alternative is to delay transfer until it does. ## Success criteria From bbcbed8eaed4d5c96be83a8122d7fbc3a9769885 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 10 Sep 2026 16:01:36 -0700 Subject: [PATCH 068/107] Fold work-log performance follow-ups into the design doc Move the three follow-ups that existed only in the work log into static_plugin_ep_registration_design.md, ahead of dropping the work log from this PR: - The web per-node cost is unattributed and needs an --enable_wasm_profiling build. Record the three-way control that already rules out the JavaScript side. - The native shared_lib A/B, its root cause (per-kernel tensor-wrapper reconstruction in the EP API adapter), the reduction candidates, the CPU-EP control that licenses the reading, and the two remaining gaps. - The plugin path emits no Api profiling events, which is an observability regression and also inverts any -p based A/B. Also qualify the web per-node figure rather than restating it. Three measurements of the same 300-node model disagree by ~3.7x (2.7, 5.0 and 10.0 us/node) and the static_lib baseline itself moved 7.90 -> 11.90 ms on the same machine within a day, so only the sign and the mechanism are well supported. A reproducible re-measurement is called out as outstanding. --- .../static_plugin_ep_registration_design.md | 59 ++++++++++++++++++- 1 file changed, 58 insertions(+), 1 deletion(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md index a0c40fd63246c..317de001af34d 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -408,6 +408,43 @@ wiring; runtime behaviour was established separately by the ORT Web prototype de A practical build note: the WASM build needs `node` on `PATH` — `node_helper.cmake` does a hard `find_program` and fails configure without it. emsdk bundles a suitable one under `cmake/external/emsdk/node//bin`. +### Native shared-library plugin performance + +The latency criterion covers two shapes of plugin, and the ORT Web numbers under +[Measured performance comparison](#measured-performance-comparison) settle only the static one. A +native `--use_webgpu shared_lib` A/B against a built-in baseline measured the plugin slower by ~8% on a +dispatch-bound model and 4-6% on Qwen3.5-0.8B, fitting roughly **67 µs per `Run` plus 1.2 µs per node**. The +reproducibility caveat recorded for the web numbers applies here too; these figures should be re-taken with the +commit, flags and harness recorded. + +One control is what licenses reading the delta as boundary cost at all: matching build flags do not prove two +executables perform alike, so both binaries were also run on the CPU EP, where no plugin is loaded. That gave a +ratio of minima of 1.001 — but only single-threaded. Run with default threading the same control showed the +plugin executable 24% *faster*, which would have been actively misleading. + +**Root cause: redundant per-kernel work, not boundary crossing.** 1.2 µs/node is ~4,400 cycles, far more than a +cross-DLL indirect call can account for, so both arms were CPU-profiled. `ep::adapter::CreateTensorFromApiValue` +rebuilds an `onnxruntime::Tensor` — nine C API round trips, an allocator-name `std::string`, a `TensorShape` +allocation — for every input and output of every node on every `Run`, because the adapter's `OpKernelContext` is +constructed fresh per `Compute()` and caches nothing beyond it. The adapter symbols are absent from the built-in +profile entirely, `Tensor`/`TensorShape` work doubles, and the extra CPU (+0.53 ms/rep) accounts for the entire +extra wall latency (+0.42 ms/rep). + +That makes the cost look **reducible rather than intrinsic**, which matters for how the latency criterion is +settled: negotiating a tolerance around a fixable defect would be the wrong order. The obvious candidates are +caching the `Tensor` wrapper across `Compute()` calls, and avoiding the allocator-name `std::string` and the +`TensorShape` allocation on the per-node path. This is also the leading candidate for the unattributed web +per-node cost, since both boundaries charge per kernel node. + +Two gaps remain in the native numbers. Session-creation cost was not compared — the plugin arm must also discover +and load the DLL, so it is likely worse in relative terms. And both arms ran with default WebGPU options; graph +capture in particular targets exactly the per-dispatch cost identified above and is worth a follow-up. + +**The plugin path silently loses the WebGPU EP's `Api` profiling events.** The built-in arm emits 15,300 `Api` +category events under `-p` and the plugin arm emits none. This is an observability regression independent of +latency, and it also means ORT's own profiler cannot perform this A/B: under `-p` the built-in arm measures +*slower* and the result inverts. + ## ORT Web migration The intended end state is that the WebGPU EP is always a plugin EP and `--use_webgpu static_lib` goes away @@ -695,7 +732,8 @@ Reading of these numbers: adapters instead of a direct in-process virtual call. Unlike the compute-bound case this one does separate cleanly — the plugin build's slowest control sample is still faster than its own fastest sample. It only shows up when nodes do essentially no GPU work, which is the worst case by construction rather than a representative - one. Amortized over ~300 nodes it is on the order of 3 µs per node. + one. Amortized over ~300 nodes it is on the order of 3 µs per node — but see the caveat below before relying on + that figure. - **Session creation costs about 3%** (measured pre-merge: 1136 → 1172 ms and 1250 → 1290 ms for the two models). This covers plugin EP registration, `GetCapability` and kernel creation through the C API. The absolute figure is dominated by fetching and parsing the model, so the true relative cost of the plugin machinery within session @@ -711,6 +749,25 @@ Two cautions on methodology: - Timings taken with `--webgpu.profiling.mode=default` are inflated by the timestamp queries and must never be compared against timings taken without it. +**The per-node magnitude is not reliably established, and a controlled re-measurement is outstanding.** Three +separate measurements of the same 300-node dispatch-bound model disagree by roughly 3.7x: +0.80 ms (2.7 µs/node) +above, +1.50 ms (5.0 µs/node) in a later four-model sweep that fitted 4.76 µs/node across 150/300/600 nodes, and ++3.00 ms (10.0 µs/node) in a four-side build comparison. The `static_lib` baseline itself moved from 7.90 ms to +11.90 ms for the same model on the same machine within a day, so build-to-build and session drift dominates the +spread, and Edge's 0.1 ms `performance.now()` floor without cross-origin isolation limits resolution further. The +three runs also did not use the same statistic (per-run P50 versus median-of-round-minimum). A re-measurement +should record the commit of each arm, the full build flag set, the model definitions, the statistic, the +interleaved sampling order and the GPU submit counts, so the number can be reproduced. + +What *is* robust is the sign and the mechanism. The plugin side was slower in 26 of 26 paired rounds in the sweep +and 6 of 6 in a three-way run, and GPU submit counts were byte-identical on every side, model and round, so the +cost is CPU-side per-kernel work rather than extra queue work. The same three-way run also excludes the +JavaScript side: a `mismatch` arm (plugin bundle against a `static_lib` wasm) came out level with `static_lib`, so +the `_OrtAppendExecutionProviderV2` call and the EP-name lookup are free and the cost is inside the +`static_plugin` wasm's per-kernel C++ path. **Where exactly is not yet attributed** — an `--enable_wasm_profiling` +build would give named frames, and the native root cause recorded under +[Native shared-library plugin performance](#native-shared-library-plugin-performance) is the leading candidate. + > Should static factories be registered before environment creation or through environment construction options? Neither. They are registered by ORT core during `OrtEnv` creation, after the environment is constructed and From 1d577c8d4ce9725e45ee3a12ba0136f5a301c9b0 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 10 Sep 2026 16:02:44 -0700 Subject: [PATCH 069/107] Drop the work log and native measurement notes from the PR These three files recorded how the branch got to its current state rather than what the current state is. Their durable conclusions and the three outstanding follow-ups now live in static_plugin_ep_registration_design.md, and no tracked file references them. They are kept on disk locally and listed in .git/info/exclude, so the detail behind the measurements is not lost even though it is not part of this PR. Note that run_native_ab.ps1 was the reproduction harness for the native A/B numbers; the outstanding re-measurement will need it or an equivalent restored. --- .../native_plugin_latency_measurements.md | 557 ----------- .../run_native_ab.ps1 | 78 -- .../static_plugin_ep_work_log.md | 897 ------------------ 3 files changed, 1532 deletions(-) delete mode 100644 docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md delete mode 100644 docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/run_native_ab.ps1 delete mode 100644 docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md deleted file mode 100644 index 8446fcec316d8..0000000000000 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/native_plugin_latency_measurements.md +++ /dev/null @@ -1,557 +0,0 @@ -# Native plugin EP latency measurements - -Tracks the native (shared-library) half of this workstream's latency criterion: - -> Inference latency stays within an accepted tolerance of the same baselines, for both static plugin registration and -> the native shared-library plugin. - -The static-plugin (wasm) half is already measured, in -[the work log](static_plugin_ep_work_log.md#measured-cost-of-the-plugin-path-on-the-web-build). This document covers -the native `--use_webgpu shared_lib` plugin against a built-in `--use_webgpu` baseline. - -Commit under test: `873f221786`. - -## Measurement environment - -| adapter | PnP status | role | -| --- | --- | --- | -| NVIDIA GeForce RTX 5060 Ti | `OK`, driver `32.0.16.1656` (616.56), 16 GB | **the measurement device** | -| NVIDIA Quadro P620 | `Error` | Pascal, stranded on the old driver — see below | -| Microsoft Remote Display Adapter | `OK` | remote-session indirect display, not a compute device | - -`nvidia-smi -L` reports the RTX 5060 Ti as the only NVIDIA GPU. This is a convenient property for an A/B: with a -single real GPU adapter present there is no ambiguity about which device either arm ran on. - -The P620 is unusable because NVIDIA ended Maxwell/Pascal/Volta support after the R570 driver branch, so the installed -616.56 package contains no kernel driver that will bind to it. Only one NVIDIA kernel driver loads at a time, so -installing a driver new enough for a Blackwell part necessarily drops the Pascal part. This is expected, not a -misconfiguration. - -### Earlier episode: measurements taken on the wrong GPU - -Recorded because it nearly corrupted this document's results, and because the same trap applies to anyone reusing -this machine. - -For part of 2026-09-04 the situation was **inverted**: the RTX 5060 Ti was in a driver-failure state -(`CM_PROB_FAILED_ADD`, Code 31) while the P620 was healthy, so every WebGPU run silently landed on the P620. Pascal -does not advertise `shader-f16`, so f16 models failed at the first node with - -``` -Program GatherBlockQuantized requires f16 but the device does not support it. -``` - -from `ORT_RETURN_IF_NOT(webgpu_context_.DeviceHasFeature(wgpu::FeatureName::ShaderF16), ...)` in -`onnxruntime/core/providers/webgpu/shader_helper.cc:421`. `WebGpuContext` only requests `ShaderF16` when the adapter -advertises it (`webgpu_context.cc:793-816`), so this was an adapter capability limit rather than a configuration -mistake. Reinstalling the driver resolved it, and both Qwen scenarios then ran clean. - -Two lessons worth keeping: - -- **A failure seen first on the plugin path is not necessarily caused by the plugin path.** The f16 failure was - reproduced identically on the built-in arm (`build/int`, `-e webgpu`), which is what identified it as - environmental. Both arms reach the same `GetAvailableRequiredFeatures()` call (`webgpu_context.cc:157`), so they - agree by construction. Always run the built-in control before attributing anything to the plugin boundary. -- **Timing runs do not announce which GPU they used.** Nothing in ORT's output names the adapter, so a machine with - more than one GPU can quietly produce numbers from the wrong one. Confirm the device externally — `nvidia-smi` - during a long run shows the process against a specific GPU — and record it alongside the numbers. - -### There is no adapter-selection knob - -`kDeviceId` (`ep.webgpuexecutionprovider.deviceId`) looks like an adapter index but is not one. It is parsed into -`WebGpuContextConfig::context_id` (`webgpu_provider_factory.cc:155-162`) and used only for context caching and -cross-device copy checks. ORT exposes no way to pick a physical adapter, so on a multi-GPU box the adapter is -whatever Dawn selects. - -**This matters for the A/B itself, not just for f16.** If the two arms could land on different adapters the -comparison is meaningless, and neither ORT nor perf_test will tell you — `--list_ep_devices` reports vendor -`Microsoft` with no GPU identity. On this machine the point is currently moot, since only one real GPU adapter is -present, but record the adapter alongside any numbers regardless. - -## Measurement setup - -### The matched build pair - -Two Ninja build directories from the same commit, differing only in the EP flag: - -``` -python tools\ci_build\build.py --config Release --build_dir D:\source\onnxruntime_4\build\nat_builtin ^ - --cmake_generator Ninja --build_shared_lib --skip_submodule_sync --parallel --update --build --skip_tests ^ - --target onnxruntime_perf_test --use_webgpu -``` - -and the same command with `--build_dir ...\build\nat_plugin --use_webgpu shared_lib`. - -Constraints that are easy to get wrong: - -- **`--build_shared_lib` is required on both arms.** The plugin DLL resolves `OrtGetApiBase` from the host, so a - statically linked host executable cannot load it. -- **Do not disable unit tests.** `onnxruntime_perf_test` is defined in `onnxruntime_unittests.cmake`; `--skip_tests` - only skips *running* them, which is what is wanted. -- **The plugin arm needs a second build step**, `ninja onnxruntime_providers_webgpu`, because perf_test `dlopen`s the - plugin and `--target onnxruntime_perf_test` alone does not produce the DLL. -- `build/int` is **not** usable as the baseline: it differs in four ways at once (RelWithDebInfo vs Release, Visual - Studio vs Ninja generator, `BUILD_SHARED_LIB=OFF` vs `ON`, and `ENABLE_DAWN_BACKEND_VULKAN=ON` vs `OFF`). It is - fine as a qualitative control, as used for the f16 result above, but not for timing. - -### Fairness control: verify the two builds differ only in the EP - -Identical `build.py` command lines do **not** guarantee identical builds, so verify the generated build rather than -trusting the invocation. - -1. Diff the two `CMakeCache.txt` files and confirm every delta is an EP-path option. Normalize the build-directory - name first, or the path embedded in most entries will swamp the diff. -2. Confirm the optimization flags actually reached the compiler, by counting them in the generated `build.ninja`: - -```powershell -$bn = 'D:\source\onnxruntime_4\build\\Release\build.ninja' -foreach ($f in '/O2','/Ob2','/DNDEBUG') { - "$f = " + (Select-String -Path $bn -Pattern $f -SimpleMatch -AllMatches | Measure-Object).Count -} -``` - -Both arms should report several thousand of each. **Zero means an unoptimized build**: MSVC defaults to `/Od` when -no `/O` flag is given, and a missing `/DNDEBUG` also leaves asserts enabled. - -This is not hypothetical — step 2 caught exactly that failure here. A first `nat_builtin` configure was interrupted -part-way through and left `CMAKE_CXX_FLAGS_RELEASE` **empty** in its cache. A later `build.py --update` reused the -damaged cache rather than regenerating it, and the build then completed with no error, producing a working but -**unoptimized** `onnxruntime_perf_test.exe`. The two arms' compile flags were otherwise character-identical: - -``` -nat_builtin: ... -DEIGEN_HAS_C99_MATH -std:c++20 -MD -Zi /GR /W4 ... -nat_plugin: ... -DEIGEN_HAS_C99_MATH /O2 /Ob2 /DNDEBUG -std:c++20 -MD -Zi /GR /W4 ... -``` - -Had this gone unnoticed it would have produced a large, entirely bogus result in the plugin's favour. The fix is to -**delete the build directory and configure from scratch** — repairing a cache in an unknown state is not worth the -risk. A corrupted cache also shows a secondary tell: stray `CMAKE_ADDR2LINE`, `CMAKE_NM`, `CMAKE_OBJCOPY`, -`CMAKE_OBJDUMP`, `CMAKE_READELF`, `CMAKE_STRIP` and `CMAKE_TAPI` entries that the healthy cache does not have. - -**Generalization: if a build directory was ever interrupted mid-configure, do not measure with it.** Interrupted -CMake configures fail silently in a way that survives `--update` and still produces a runnable binary. - -### Invocations - -Built-in arm: - -``` -onnxruntime_perf_test.exe -e webgpu -m times -r \model.onnx -``` - -Plugin arm: - -``` -onnxruntime_perf_test.exe --plugin_ep_libs "WebGPU|onnxruntime_providers_webgpu.dll" ^ - --plugin_eps WebGpuExecutionProvider -m times -r \model.onnx -``` - -- The EP name is **`WebGpuExecutionProvider`**, not `WebGPU`. The registration name before the `|` is arbitrary. -- perf_test **rejects a model directory** — pass the `.onnx` file. It picks up `test_data_set_0` from that directory. -- `-i` is ignored for WebGPU: `-e webgpu` hardcodes `AppendExecutionProvider("WebGPU", {})` - (`ort_test_session.cc:608-613`). Use **`-C "ep.webgpuexecutionprovider.|"`**, which routes through - `ConfigOptions` and therefore applies **identically on both arms** — exactly what an A/B needs. The available keys - are in `webgpu_provider_options.h`. -- **Do not gate on the process exit code.** perf_test returns non-zero even on a fully successful run, because ORT's - memory-leak checker reports a handful of 16-byte CRT static-initializer allocations (`initterm`) at shutdown. The - report is emitted after timing and affects both arms identically, so it does not bias the comparison — but scripts - should parse `Min Latency` from stdout rather than trusting the exit status. - -Methodology: interleaved rounds, taking the median of each round's minimum — the same procedure used for the wasm -numbers, so the two sets are directly relatable. - -Note that perf_test **cannot drive a *statically* linked plugin EP**: both `common_utils.cc:93-99` and -`ort_test_session.cc:111-112` gate on `registered_plugin_eps`, which is populated only by `--plugin_ep_libs`. That is -moot for this dynamic-library A/B, but it is a real gap if static-plugin native perf is ever needed. - -### Models - -`bench_compute` (16 nodes) and `bench_dispatch` (300 nodes) are fp32. `bench_dispatch` is the -dispatch-overhead-sensitive case, and is where the wasm A/B found the per-kernel cost. - -A Qwen3.5-0.8B int4 case is also included, to check whether the per-kernel overhead is visible on a real LLM. Both -its scenarios have been confirmed to run end to end on the RTX 5060 Ti. - -#### Why the Qwen inputs had to be generated by hand - -perf_test's `-I` flag cannot produce valid inputs for this model. `InitializeTensorWithSeed` -(`ort_test_session.cc:1020-1040`) randomizes only `float` and `int8_t`/`uint8_t`; every other type falls through to -`random_init = false` and prints "this type of data won't be random initialized". `input_ids` and `position_ids` are -`int64` and would be left **uninitialized**, indexing far outside the 248320-entry embedding table. Unspecified free -dimensions also default to 1 (`:1135-1137`), which would give a nonsense shape regardless. - -So a real `test_data_set_0` was generated instead. Tensor **names** are set on every `TensorProto`, which makes the -lexicographic `input_10.pb` < `input_2.pb` file ordering harmless — `TestCase.cc:621-624` prefers the name over the -positional index whenever the name is non-empty. - -The model is hybrid-attention (`qwen3_5_text`: 24 layers, vocab 248320, hidden 1024, 8 heads, 2 KV heads, -head size 256), so its 51 inputs are not uniform: - -| input | index | type | shape | -| --- | --- | --- | --- | -| `input_ids` | 0 | int64 | `[B, S]` | -| `attention_mask` | 1 | int64 | `[B, T]` | -| `past_key_values.{L}.key`, L in {3,7,11,15,19,23} | 2-7 | float16 | `[B, 2, P, 256]` | -| `past_key_values.{L}.value`, same L | 8-13 | float16 | `[B, 2, P, 256]` | -| `position_ids` | 14 | int64 | `[B, S]` | -| `past_key_values.{L}.conv_state`, the other 18 layers | 15-50 | float16 | `[B, 6144, 3]` | -| `past_key_values.{L}.recurrent_state`, same 18 layers | 15-50 | float16 | `[B, 16, 128, 128]` | - -Only 6 of the 24 layers carry a KV cache; the remaining 18 are linear-attention layers with convolution and -recurrent state. Two scenarios were materialized under `D:\test\qwen_perf`: `decode` (B=1, S=1, P=128, T=129, the -dispatch-sensitive case) and `prefill` (B=1, S=128, P=0, T=128, a compute-bound control). - -`genai_config.json` sets `past_present_share_buffer: true` and WebGPU options `enableGraphCapture: "0"` / -`validationMode: "basic"`. perf_test does not apply these, so pass them with `-C` if the comparison should match how -GenAI runs the model. - -## Results - -Measured on the RTX 5060 Ti, on an otherwise idle machine. The two `build.py` configures differ by exactly one -generator flag, `-Donnxruntime_USE_EP_API_ADAPTERS=ON` on the plugin arm. - -### Checks performed before measuring - -| check | result | -| --- | --- | -| Normalized `CMakeCache.txt` diff | **2 lines, one key**: `onnxruntime_USE_EP_API_ADAPTERS` `OFF` vs `ON` | -| `/O2` / `/Ob2` / `/DNDEBUG` in `build.ninja` | 4184 built-in, 4183 plugin — off-by-one is WebGPU built in-tree | -| Binaries current with the tree | every commit since `873f221786` is doc-only, so the code under test is HEAD's | - -### The plugin arm really is going through the plugin - -A latency number is meaningless if the "plugin" arm quietly fell back to something else, so this was established by -making the alternatives fail rather than by reading the code: - -| invocation on the plugin build | outcome | -| --- | --- | -| `-e webgpu` (built-in path) | fails: "WebGPU execution provider is not supported in this build" | -| `--plugin_eps BogusEpName` | fails: "[Plugin EP]: No matching EP devices found." (`common_utils.cc:171`) | -| `--plugin_eps WebGpuExecutionProvider` | runs | - -The first line is the important one: WebGPU is genuinely absent from this build's `onnxruntime.dll`, so a run that -succeeds can only have come from `onnxruntime_providers_webgpu.dll`. The second confirms the EP-name lookup is real -and not silently permissive. - -### Control: the two binaries are equivalent when no plugin is involved - -Matching build flags still do not prove the two executables perform alike. Running both on the **CPU EP**, where -neither arm loads a plugin, isolates the plugin boundary from any residual build difference. - -Run with default threading this control was actively misleading — `bench_compute` came out **24% faster** on the -plugin executable (4.61 ms vs 3.50 ms), which is not a real effect in any direction and simply shows how noisy the -multi-threaded CPU EP is here. Repeating it single-threaded (`-x 1 -y 1`, 10 rounds) removes the thread-pool -variance: - -| `bench_compute`, CPU EP, single-threaded | min | median | max | -| --- | --- | --- | --- | -| built-in build | 20.011 ms | 20.285 ms | 20.734 ms | -| plugin build | 20.023 ms | 20.499 ms | 20.848 ms | - -Ratio of minima **1.001**, medians within 1%. The two executables are performance-equivalent absent a plugin, so the -WebGPU differences below are attributable to the plugin boundary rather than to the builds. - -### WebGPU A/B - -8 interleaved rounds; `-r 30` for the bench models, `-r 20` for Qwen. "median" is the median of the per-round -minimums, matching the wasm procedure; "best" is the single fastest round, which is the most noise-resistant -statistic available. - -| model | nodes | built-in median | plugin median | delta | ratio | best-of ratio | rounds plugin slower | -| --- | --- | --- | --- | --- | --- | --- | --- | -| `bench_compute` | 16 | 1.662 ms | 1.917 ms | +0.255 ms | 1.153 | 1.054 | 8/8 | -| `bench_dispatch` | 300 | 5.118 ms | 5.562 ms | +0.444 ms | 1.087 | 1.084 | 8/8 | -| Qwen decode | 1043 | 34.95 ms | 36.95 ms | +1.99 ms | 1.057 | 1.092 | 6/8 | -| Qwen prefill | 1043 | 47.66 ms | 49.57 ms | +1.92 ms | 1.040 | 1.064 | 6/8 | - -**The plugin EP is consistently slower.** Raw per-round minimums, in milliseconds, in round order: - -| model | arm | per-round minimums | -| --- | --- | --- | -| `bench_compute` | built-in | 1.653, 1.626, 1.686, 1.661, 1.895, 1.877, 1.662, 1.614 | -| `bench_compute` | plugin | 1.700, 1.862, 2.055, 1.702, 1.971, 2.189, 1.701, 2.345 | -| `bench_dispatch` | built-in | 5.170, 5.131, 5.105, 5.158, 5.112, 5.124, 5.111, 5.104 | -| `bench_dispatch` | plugin | 5.577, 5.544, 5.623, 5.584, 5.543, 5.546, 6.091, 5.533 | -| Qwen decode | built-in | 31.34, 34.90, 40.56, 36.13, 35.01, 38.00, 30.57, 34.63 | -| Qwen decode | plugin | 35.32, 40.86, 43.65, 34.24, 36.33, 33.40, 39.88, 37.56 | -| Qwen prefill | built-in | 43.05, 49.42, 44.24, 50.92, 47.75, 47.56, 46.17, 48.23 | -| Qwen prefill | plugin | 49.99, 49.56, 50.59, 52.32, 48.80, 45.80, 49.58, 45.89 | - -### How much of this is signal - -The four rows are not equally trustworthy, and the medians alone hide that: - -- **`bench_dispatch` is the reliable result.** The built-in arm spans 5.104–5.170 ms — a 66 µs range — and the - per-round delta is +0.41 to +0.52 ms in seven of eight rounds (the eighth, +0.98 ms, is a visible outlier). An - effect of +0.43 ms against that spread is unambiguous. -- **`bench_compute` is noisier than its median suggests.** Per-round deltas range from +0.039 to +0.731 ms. The - plugin arm's slow rounds inflate the median to 1.153; the best-of ratio of 1.054 is the more defensible figure. -- **Both Qwen rows are noise-dominated.** Per-round deltas swing from −4.6 to +9.3 ms, and the sign flips in 2 of 8 - rounds. The direction agrees with the microbenchmarks on both the median and the best-of statistic, but these - numbers should be read as "consistent with a few percent" and not as a measurement of it. - -### Mechanism - -Fitting the two microbenchmarks' best-of deltas as a fixed per-`Run` cost plus a per-node cost gives roughly -**67 µs per `Run` + 1.2 µs per node**. That is a two-point fit and should be treated as an order-of-magnitude model -only, but it does predict outside its fitting range: for Qwen's 1043 nodes it gives ≈1.3 ms, against an observed -median delta of ≈1.9–2.0 ms — the right magnitude, and under-predicting by about 1.5x. Per-node dispatch cost is -therefore a plausible dominant term, which is what the shape of the boundary would predict: each kernel dispatch -crosses the `OrtEp` C API adapter into a separate DLL instead of making an in-module C++ virtual call. - -Compared with the wasm static-plugin A/B, which found ≈4.8 µs/node -([work log](static_plugin_ep_work_log.md#measured-cost-of-the-plugin-path-on-the-web-build)), the native -shared-library boundary costs roughly **1.2–1.5 µs/node**, some 3–4x less per node. - -### Assessment against the criterion - -A per-node cost means the overhead is a function of graph size, not a fixed tax, so no single percentage -characterizes it. On the real-model cases it lands at **4–6%**, and it grows as models get more numerous, smaller -kernels. Whether that is "within an accepted tolerance" is a judgement call that has not yet been made — the -tolerance has never been quantified, and it should be, since these results show the answer is not "zero overhead". - -Two caveats on scope: - -- Only steady-state `Min Latency` was compared. Session-creation cost, where the plugin must additionally discover - and load the DLL, was not measured and is likely worse in relative terms. -- Both arms ran with default WebGPU options. `genai_config.json` uses `enableGraphCapture: "0"` and - `validationMode: "basic"`; graph capture in particular could change the per-dispatch picture substantially, and is - worth a follow-up since it targets exactly the cost identified here. - -Reproduce with `run_native_ab.ps1` (parses `Min Latency`, ignores the exit code); raw rows land in -`native_ab_raw.csv`. - -## Where the per-node cost goes (CPU profile) - -The 1.2 µs/node figure above is far too large to be call-transport cost — a cross-DLL indirect call through the C API -is tens of cycles, while 1.2 µs at 3.696 GHz is ≈4,400 cycles. That gap motivated profiling the two arms to find out -what the cycles are actually spent on. The answer is that they are spent **host-side, not on crossing the boundary**. -Part of that host-side work is per-kernel marshalling that the built-in path does not do at all — but only a minority -part; see [Follow-up: testing the adapter hypothesis](#follow-up-testing-the-adapter-hypothesis). - -### Method - -Both arms were relinked with `/DEBUG` to produce PDBs. This was a link-only change: `LINK_FLAGS` already carried an -explicit `/OPT:REF,ICF,LBR`, so adding `/DEBUG` does not fall back to the `/OPT:NOREF,NOICF` debug defaults, and no -source file was recompiled. Binary sizes were unchanged and the A/B ratio was re-measured at 1.087 afterwards, so the -profiled binaries are the same ones the latency numbers came from. - -Two profilers were tried and rejected before settling on a third: - -- **ORT's own `-p` profiler cannot be used for this comparison.** Instrumentation is asymmetric — the built-in arm - emits 15,300 `Api` category events and the plugin arm emits **zero**, so under `-p` the built-in arm measures - *slower* (0.772 s vs 0.644 s over 50 runs), reversing the true result. It also inflates per-node time from ~17 µs - to ~46 µs by forcing per-node synchronisation. One result did survive: **per-op median kernel time is identical on - both arms (29.0 µs for Add/Mul/Sub)**, establishing early that GPU-side work is unchanged and the cost is host-side. - That the plugin path silently loses the WebGPU EP's `Api` profiling events is itself an observability regression. -- **ETW/xperf resolved no symbols.** Traces collected fine (355k+ stack samples), but every frame rendered as - `***unknown***` — zero `Module!Func` tokens in the entire report — despite symcache files being generated for all - three PDBs, and with or without the Microsoft symbol server on the path. -- **A purpose-built sampling profiler** (`sampler.cpp`) was used instead: it launches the target, samples every thread's - instruction pointer at ~1 kHz, and symbolizes leaf addresses with dbghelp against the local PDBs. Needs no - elevation, and symbolizes ORT frames reliably. - -Runs were 6000 reps of `bench_dispatch` (300 nodes), discarding the first 3 s, with the two arms pooled over two -repetitions. Sampling did not destroy the effect being studied: the latency ratio under the profiler was 1.083, -1.081 and 1.083 across three runs, against 1.087 unsampled. - -### Result - -Samples are counted only when a thread is actually running (leaf address not in a wait stub). One sample corresponds -to roughly one sampler loop period, ~1.0–1.2 ms of thread time, so **absolute µs below are approximate to about -±20%; the relative comparisons are not affected.** - -| Scope | built-in | plugin | delta | ratio | -| --- | ---: | ---: | ---: | ---: | -| All modules, running samples/rep | 3.388 | 3.922 | +0.534 | 1.158 | -| `onnxruntime.dll` + `onnxruntime_providers_webgpu.dll` only | 1.175 | 1.441 | +0.267 | 1.227 | - -Per-module numbers are meaningless on their own here, because the WebGPU code simply *moves* out of `onnxruntime.dll` -into `onnxruntime_providers_webgpu.dll`; only the combined ORT-side total is comparable. - -The two totals cross-check the latency result. The extra CPU of +0.534 samples/rep is ≈0.53–0.64 ms/rep against a -measured extra wall latency of 0.42 ms/rep, and the ORT-side share alone (+0.267 samples/rep ≈ 0.9–1.1 µs/node) is -close to the 1.2 µs/node from the latency fit. **The overhead is host-side CPU work, and most of it is ORT-side.** - -Attributing the +0.534 samples/rep by what the code is doing: - -| Category | built-in | plugin | delta | share of delta | -| --- | ---: | ---: | ---: | ---: | -| other (mostly GPU driver, see below) | 1.752 | 1.952 | +0.199 | 37.3% | -| alloc/free | 0.573 | 0.669 | +0.096 | 17.9% | -| `Tensor`/`TensorShape` construction | 0.083 | 0.165 | +0.082 | 15.3% | -| EP API adapter | 0.002 | 0.074 | +0.072 | 13.4% | -| dawn (refcounting, error checks) | 0.495 | 0.549 | +0.054 | 10.1% | -| webgpu kernel bodies | 0.173 | 0.208 | +0.034 | 6.4% | -| ort framework | 0.128 | 0.118 | −0.011 | −2.0% | - -The `ep-adapter` and `Tensor`/`TensorShape` rows are the reproducible core: they were +0.060/+0.063 and +0.030/+0.032 -in the two individual runs, i.e. stable, and the adapter work is *new* — it is ~0.002 in the built-in arm, meaning it -does not exist there. The `dawn`, `webgpu-kernel` and `alloc/free` rows moved noticeably between the two runs and -should be read as directional only. - -These symbols appear in the plugin arm and are absent from the built-in arm: - -| Symbol | plugin samples/rep | -| --- | ---: | -| `onnxruntime::TensorShape::TensorShape` | 0.020 | -| `onnxruntime::webgpu::ComputeContext::Input` | 0.016 | -| `onnxruntime::ep::adapter::CreateTensorFromApiValue` | 0.013 | -| `Ort::GetApi` | 0.013 | -| `onnxruntime::TensorShape::Allocate` | 0.013 | -| `onnxruntime::PluginEpOpKernel::Compute` | 0.008 | -| `onnxruntime::Tensor::operator=` | 0.007 | -| `onnxruntime::ep::adapter::KernelImpl::ComputeImpl` | 0.006 | -| `OrtApis::IsTensor` | 0.006 | -| `OrtApis::GetTensorElementTypeAndShapeDataReference` | 0.005 | -| `onnxruntime::DataTypeImpl::TensorTypeFromONNXEnum` | 0.004 | - -### Mechanism - -The profile points at `ep::adapter::CreateTensorFromApiValue` -([`include/onnxruntime/ep/adapter/tensor_helper.h`](../../../../include/onnxruntime/ep/adapter/tensor_helper.h)), -which rebuilds an `onnxruntime::Tensor` from an `OrtValue` on every access. Each call makes roughly nine C API round -trips (`IsTensor`, `GetTensorElementTypeAndShapeDataReference`, `GetTensorMemoryInfo`, then `GetAllocatorName`, -`GetAllocatorType`, `GetDeviceType`, `GetMemoryType`, `GetVendorId`, `GetDeviceId`), allocates a `std::string` for -the allocator name, constructs a `TensorShape` (a heap allocation once the rank exceeds the inline capacity), looks -up `DataTypeImpl::TensorTypeFromONNXEnum`, and constructs a `Tensor`. - -The adapter `OpKernelContext` ([`op_kernel.h`](../../../../include/onnxruntime/ep/adapter/op_kernel.h)) is built -fresh per `Compute()` call and `resize()`s an input and an output `std::vector`. Its tensor cache lives only -for the duration of that one call, so the work above repeats for every input and every output of every node on every -`Run`. The built-in path does none of this: it receives `const Tensor*` directly from the executor. - -The overhead is **not** the cost of crossing the DLL boundary: it is host-side work, which is why it is ~4,400 -cycles rather than the tens of cycles an indirect cross-module call would cost. Redundant per-kernel wrapper -construction and its allocation traffic are a real and *new* part of that work. - -> **Read [Follow-up: testing the adapter hypothesis](#follow-up-testing-the-adapter-hypothesis) before acting on -> this section.** An attempt to remove part of this adapter work produced **no measurable latency improvement**. The -> attribution table above is not contradicted by that result — it already assigns the adapter only 13.4% of the -> delta — but the wording of this section originally implied the adapter was the dominant cost, which the table does -> not support. - -### Caveats - -- Part of the delta is plausibly *consequence* rather than cause. The `other` bucket is dominated by GPU driver - modules (`nvwgf2umx.dll` +0.085, `win32u.dll` +0.017, `D3D12Core.dll` +0.014), and driver threads that poll will - accumulate samples in proportion to wall time — which the plugin arm has more of. The unambiguous causal signal is - the adapter and tensor-construction work, which is absent from the built-in arm entirely. -- Driver and `ntdll` frames symbolize to the nearest export (no PDBs available), so names such as - `RtlCreateUnicodeString+0x118f` identify a region, not the actual function. Module attribution is still sound. -- Suspend/resume sampling perturbs both arms, but equally; the preserved latency ratio is the check on this. -- This is a flat (leaf-address) profile, so it attributes self time only and does not give call trees. - -### Implication for the tolerance question - -Open item 8 in the work log asks what tolerance is acceptable. The first reading of these results was that the -dominant cost is redundant adapter work and therefore looks reducible, so the tolerance question should wait. - -**That reading did not survive being tested.** The adapter is a minority cost by this profile's own attribution -(13.4% of the delta, or 28.7% if the `Tensor`/`TensorShape` row is counted as adapter-caused), and an attempt to -remove part of it changed nothing measurable. See the next section. - - -## Follow-up: testing the adapter hypothesis - -The section above concluded that the per-node overhead is redundant adapter work. That hypothesis was then tested -directly by removing a measurable part of that work and re-running the same A/B. **It produced no measurable latency -improvement.** This section records the experiment, because the negative result is what stops the next person from -repeating it. - -### The change that was tried - -`CreateTensorFromApiValue` rebuilds an `OrtMemoryInfo` on every tensor access, costing six C API round trips -(`GetAllocatorName`, `GetAllocatorType`, `GetDeviceType`, `GetMemoryType`, `GetVendorId`, `GetDeviceId`) plus a -`std::string` copy. Every input and output of a node normally lives on the same device, so almost all of these -reconstructions are identical. - -The change memoized them in the adapter `OpKernelContext`, keyed on the `const OrtMemoryInfo*` returned by -`GetTensorMemoryInfo`, and also removed a duplicated `GetMemoryType()` call. The cache must be scoped to a single -`Compute()` call: `OrtApis::GetTensorMemoryInfo` returns `&tensor.Location()`, a pointer *into the tensor*, so the -pointer identifies an allocator only while that tensor is alive. A session-lifetime cache would silently return -wrong memory info once an address was recycled. - -### The change worked, mechanically - -Verified with the same sampling profiler, 30,000 reps of `bench_dispatch`, shares normalized against total samples -(the sampler's tick rate differs between runs, so raw per-rep counts are not comparable across runs): - -| Quantity | before | after | change | -| --- | ---: | ---: | ---: | -| `OrtApis::GetTensorMemoryInfo` | — | — | −78% | -| `OrtApis::MemoryInfoGetDeviceType` | — | — | −73% | -| memory-info line, share of `CreateTensorFromApiValue` | 43.3% | 29.4% | −57% of total | -| `CreateTensorFromApiValue` self time | 8.93e-5 | 5.70e-5 | −36% | -| `KernelImpl::ComputeImpl` self time (control) | 8.53e-5 | 8.91e-5 | unchanged | - -### It made no measurable difference to latency - -All `bench_dispatch` runs, same session, same machine state, built-in arm untouched as a control: - -| run | harness | built-in | plugin | delta | ratio | -| --- | --- | ---: | ---: | ---: | ---: | -| baseline | dedicated, 12 rounds | 5.1656 | 5.6205 | +0.4549 | 1.0881 | -| baseline | dedicated, 16 rounds | 5.1253 | 5.5671 | +0.4418 | **1.0862** | -| baseline | 4-model, 8 rounds | 5.1912 | 5.6805 | +0.4893 | 1.094 | -| with cache | dedicated, 12 rounds | 5.1147 | 5.5543 | +0.4397 | **1.0860** | -| with cache | 4-model, 8 rounds | 5.1710 | 5.5353 | +0.3643 | 1.070 | - -The like-for-like pair — the two dedicated runs, same day, same harness — is **1.0862 against 1.0860, a dead heat**. -The one encouraging number (1.070) came from the 4-model harness, whose own baseline was simultaneously its worst -reading (1.094); the spread of that row across runs (delta 0.364–0.489 ms) is larger than the effect being looked for. - -Across all four models, with a same-session baseline: - -| model | baseline delta / ratio | with cache delta / ratio | -| --- | --- | --- | -| `bench_compute` | +0.0515 / 1.029 | +0.0439 / 1.027 | -| `bench_dispatch` | +0.4893 / 1.094 | +0.3643 / 1.070 | -| Qwen decode | +1.5917 / 1.050 | +2.1669 / 1.068 | -| Qwen prefill | +0.3782 / 1.009 | +0.5557 / 1.013 | - -Two rows improved and two got worse. Both Qwen rows are noise-dominated, as established earlier in this document, so -the mixed signs are expected and carry no information. - -Note that the earlier committed table cannot be used as the baseline here: a GPU driver update moved the built-in -control itself (Qwen decode 34.95 → 31.79 ms, prefill 47.66 → 43.03 ms). **Any future A/B on this branch must -re-measure its own baseline in the same session rather than comparing against numbers recorded in this document.** - -### Why: the addressable budget is small - -Converting sampler counts to time on the critical thread explains the null result. One sample is one tick of one -thread, so a symbol's time per rep is `samples / sampled_wall_ms * rep_ms`. The sampler samples *every* thread each -tick — around 25 threads here, most of them parked in wait stubs — so normalizing by total samples or by rep count -rather than by sampled wall time overstates or understates a symbol by the thread count. Adapter and C API code runs -on the single inference thread, so for those symbols this conversion is directly comparable to latency. - -| Scope | `ep::adapter::*` | `OrtApis::*` + `Ort::GetApi` | sum | -| --- | ---: | ---: | ---: | -| built-in (control) | 0.0 | 9.2 | 9.2 µs/rep | -| plugin, before the change | 26.7 | 41.2 | 67.9 µs/rep | -| plugin, after the change | 29.3 | 34.9 | 64.1 µs/rep | - -So the entire adapter plus C API layer costs about **59 µs/rep more than built-in, against a total gap of ~455 -µs/rep — roughly 13%.** That is the same 13.4% the attribution table in the previous section already reported. Even -perfect elimination of all adapter and C API cost cannot close more than about one eighth of the deficit, and the -change actually made addressed only part of that eighth. An effect of that size is below this setup's noise floor, -so the experiment does not prove the change is worthless — it bounds it at under roughly 5% of the gap. - -The change was reverted rather than kept: it adds a cache class and a non-obvious lifetime rule for no demonstrated -benefit. - -### What this leaves - -The adapter is a real, new, and reducible cost, but it is a minority one. The remaining ~85% of the gap is -elsewhere. The plugin-minus-built-in profile points at `dawn` refcounting, `std::string`, `TensorShape`, and -`ComputeContext::Input` all being heavier in the plugin build — a pattern consistent with **lost cross-module -inlining** once the WebGPU code moves into its own DLL, which no amount of adapter tuning can recover. Candidates -worth testing before revisiting the adapter: - -- link-time code generation / whole-program optimization on the plugin DLL, to test the lost-inlining theory; -- graph capture (`enableGraphCapture`), which targets per-dispatch cost directly and has never been measured here; -- confirming whether the GPU-driver share of the delta (`nvwgf2umx.dll` +0.085 samples/rep) is genuine extra work or - a wall-time-proportional polling artifact — i.e. consequence rather than cause. - -A caveat that applies to all of the above: this is still a flat, leaf-address profile. It measures self time and -cannot prove that a `malloc` or `dawn` sample was reached *from* any particular caller. More samples improve -precision, not attribution. Settling that would need stack walking, which has not been implemented. \ No newline at end of file diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/run_native_ab.ps1 b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/run_native_ab.ps1 deleted file mode 100644 index fb42062576ab1..0000000000000 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/run_native_ab.ps1 +++ /dev/null @@ -1,78 +0,0 @@ -# Interleaved A/B: built-in WebGPU EP vs shared-library plugin EP. -# Parses "Min Latency" from stdout; perf_test's exit code is unreliable because the -# shutdown leak checker reports benign CRT static-init allocations. -param( - [int]$Rounds = 8, - [int]$Reps = 30, - [int]$QwenReps = 20, - [string]$OutCsv = 'D:\test\native_ab_raw.csv' -) - -$builtinExe = 'D:\source\onnxruntime_4\build\nat_builtin\Release\onnxruntime_perf_test.exe' -$pluginDir = 'D:\source\onnxruntime_4\build\nat_plugin\Release' -$pluginExe = "$pluginDir\onnxruntime_perf_test.exe" - -$models = [ordered]@{ - 'bench_compute' = 'D:\source\onnxruntime_4\js\web\test\data\bench\bench_compute\model.onnx' - 'bench_dispatch' = 'D:\source\onnxruntime_4\js\web\test\data\bench\bench_dispatch\model.onnx' - 'qwen_decode' = 'D:\test\qwen_perf\decode\model.onnx' - 'qwen_prefill' = 'D:\test\qwen_perf\prefill\model.onnx' -} - -function Get-MinLatency([string]$text) { - $m = [regex]::Match($text, 'Min Latency:\s*([0-9.eE+-]+)\s*s') - if ($m.Success) { return [double]$m.Groups[1].Value } - return $null -} - -function Invoke-Arm([string]$arm, [string]$model, [int]$reps) { - if ($arm -eq 'builtin') { - $out = & $builtinExe -e webgpu -m times -r $reps $model 2>&1 | Out-String - } else { - Push-Location $pluginDir - try { - $out = & $pluginExe --plugin_ep_libs "WebGPU|onnxruntime_providers_webgpu.dll" ` - --plugin_eps WebGpuExecutionProvider -m times -r $reps $model 2>&1 | Out-String - } finally { Pop-Location } - } - return (Get-MinLatency $out) -} - -function Get-Median([double[]]$v) { - $s = $v | Sort-Object - $n = $s.Count - if ($n -eq 0) { return $null } - if ($n % 2 -eq 1) { return $s[[int](($n-1)/2)] } - return ($s[$n/2 - 1] + $s[$n/2]) / 2 -} - -$rows = @() -foreach ($r in 1..$Rounds) { - foreach ($name in $models.Keys) { - $reps = if ($name -like 'qwen*') { $QwenReps } else { $Reps } - # Interleave within the round so slow drift affects both arms equally. - $b = Invoke-Arm 'builtin' $models[$name] $reps - $p = Invoke-Arm 'plugin' $models[$name] $reps - $rows += [pscustomobject]@{ - round = $r; model = $name - builtin_ms = if ($b -ne $null) { [math]::Round($b*1000,4) } else { $null } - plugin_ms = if ($p -ne $null) { [math]::Round($p*1000,4) } else { $null } - } - Write-Host ("round {0} {1,-15} builtin={2,9:N4} ms plugin={3,9:N4} ms" -f $r,$name,($b*1000),($p*1000)) - } -} - -$rows | Export-Csv -NoTypeInformation -Path $OutCsv -Write-Host "`n=== raw rows written to $OutCsv ===" -Write-Host "`n=== SUMMARY (median of per-round minimums) ===" -Write-Host ("{0,-15} {1,12} {2,12} {3,10} {4,9} {5,13}" -f ` - 'model','builtin_ms','plugin_ms','delta_ms','ratio','plugin_slower') -foreach ($name in $models.Keys) { - $sub = $rows | Where-Object { $_.model -eq $name -and $_.builtin_ms -ne $null -and $_.plugin_ms -ne $null } - if (-not $sub) { Write-Host ("{0,-15} NO DATA" -f $name); continue } - $mb = Get-Median ([double[]]($sub.builtin_ms)) - $mp = Get-Median ([double[]]($sub.plugin_ms)) - $slower = ($sub | Where-Object { $_.plugin_ms -gt $_.builtin_ms }).Count - Write-Host ("{0,-15} {1,12:N4} {2,12:N4} {3,10:N4} {4,9:N3} {5,10}/{6}" -f ` - $name, $mb, $mp, ($mp-$mb), $(if($mb -gt 0){$mp/$mb}else{0}), $slower, $sub.Count) -} diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md deleted file mode 100644 index d05288b35a8f8..0000000000000 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_work_log.md +++ /dev/null @@ -1,897 +0,0 @@ -# Static plugin EP registration — work log - -Working notes for the branch `edgchen1/webgpu_static_plugin`. This is a scratch handoff document, not a -design document: the design and its rationale live in -[`static_plugin_ep_registration_design.md`](./static_plugin_ep_registration_design.md), which is the -authoritative reference for decisions D1–D7. **Delete this file before opening the PR.** - -The purpose of this log is to let someone pick the work up on a different machine — in particular a machine -with a real GPU, which the original development machine did not have. - -## Goal - -Make the statically linked WebGPU EP build go through the **plugin EP** path rather than the internal-EP path, -via a generic "static plugin EP" registration mechanism in ORT core. The same EP sources -(`onnxruntime/core/providers/webgpu/ep/*`) then serve both a dynamically loaded plugin DLL and a build that is -statically linked into the host. The latter is what unblocks ORT Web / Emscripten, where shared library modules -are not possible. - -Whether the EP is dynamically or statically **linked** is deliberately separate from whether it is a **plugin**. -Both linkage modes are supported. - -## Status - -The implementation is complete and committed. CI coverage was added for the new configuration, which had none. - -| Area | State | -| --- | --- | -| Implementation | Done | -| Windows build + GPU tests (`--use_webgpu static_plugin`) | Green in CI | -| Shared library plugin build (`--use_webgpu shared_lib`) | Green in CI, no regression from D2/D7 | -| CUDA plugin EP (shares `include/onnxruntime/ep/*`) | Green in CI | -| Operator-level validation on a real GPU | **Done — no behavioural difference**, see [Testing](#testing) | -| Linux/GCC build (`--use_webgpu static_plugin`) | **Green in CI** (run `33030694040`) after the `array-bounds` fix — see [GCC](#gcc-and-warnings-as-errors) | -| Minimal build | Not yet exercised — see [Open items](#open-items) | -| Emscripten / ORT Web | **Builds, links, and runs in a browser** on a real GPU; registration verified in the binary *and* at runtime — see [ORT Web](#emscripten-and-ort-web) | -| Latency | **Measured and root-caused** — ~4.8 us/node (wasm), ~1.2 us/node (native); the native cost is redundant per-kernel tensor-wrapper construction in the EP API adapter, not boundary-crossing cost, so it looks reducible — see [Open items](#open-items) | -| PR | Draft [#32395](https://github.com/microsoft/onnxruntime/pull/32395); self-review feedback addressed — see [PR review feedback](#pr-review-feedback) | - -## Branch - -- Branch: `edgchen1/webgpu_static_plugin`, pushed to `git@github.com:microsoft/onnxruntime.git` (not a fork — - this matters, because the self-hosted 1ES CI pools are not available to forks). -- Forked from `main` at `fab4691797560a22092d9bbe8cec540b5413724b`. -- The earlier commits on the branch (`dff9cc42bf` .. `800ad881ef`) are the WebGPU extraction planning documents - and are unrelated to this workstream's code. The code work starts at `2754674229`. - -| Commit | Summary | -| --- | --- | -| `2754674229` | Support registering statically linked plugin EPs, and build WebGPU EP that way | -| `32775c305a` | Add CI jobs for the WebGPU static plugin EP build | -| `49855f6ef7` | Fix GCC maybe-uninitialized errors in the EP adapter attribute accessors | -| `d47f00a0a6` | Update dynamic plugin EP test config parsing tests | -| `1980e901ec` | Record CI coverage in the static plugin EP registration design doc | -| `754e025ddc` | Add a work log for the static plugin EP branch | -| `a362107df9` | Merge `origin/main` (52 commits behind at the time; merged, not rebased) | -| `35e6f65d52` | Record real-GPU validation results in the work log | -| `a110948b1d` | Record GPU validation and the plugin-path test coverage gap in the design doc | -| `8d87c11972` | Avoid GCC 14 `-Warray-bounds` false positive in `EmitGateActivationExpr` — the batch-3 CI fix | -| `ef68884a95` | Record the complete GCC 14 warning inventory and the local reproduction recipe | - -Pushed through `ef68884a95`. The push from `754e025ddc` was a fast-forward; no history has been rewritten on -this branch. - -## What the change consists of - -Read the design doc first. The short version, keyed to the decisions: - -- **D1** — A link-time registry in ORT core (`onnxruntime/core/session/plugin_ep/ep_static_plugins.{h,cc}`), - hand-written and `#if`-guarded, mirroring `EpLibraryInternal::CreateInternalEps`. Code generation from CMake - and a new public C API were both considered and rejected. -- **D2** — `ORT_PLUGIN_EP_ENTRY_POINT_PREFIX` in `onnxruntime/core/providers/webgpu/ep/api.cc`. The shared - library build must keep exporting **unprefixed** `CreateEpFactories` / `ReleaseEpFactory`, because - `EpLibraryPlugin::Load` looks those up by name. The static build emits `WebGpu_`-prefixed symbols instead. -- **D3** — No new ABI hook for process-global teardown. `ShutdownProtobufLibrary()` is compiled out unless - `ORT_PLUGIN_EP_OWNS_PROCESS_GLOBALS`. The legacy WebGPU cleanup block in `ort_env.cc` must stay for the - default internal-EP build. -- **D4** — CMake option `onnxruntime_WEBGPU_STATIC_PLUGIN` plus the derived - `onnxruntime_WEBGPU_LINKED_INTO_HOST`; surfaced as `--use_webgpu static_plugin`. -- **D5** — `ep_library_path` became optional in the dynamic plugin EP test infrastructure's - `InitializationConfig`, so a statically registered EP can be selected without a path to load. -- **D6** — Publish-then-register in `ort_env.cc`, and `OrtEnv::m_` became a `std::recursive_mutex`. Registration - reenters the environment (`GetSupportedDevices()` → `Api().ep.GetEnvConfigEntries()` → - `OrtEnv::TryGetInstance()`), which self-deadlocked with a plain mutex. -- **D7** — `ORT_PLUGIN_EP_STATICALLY_LINKED` disables the manual C++ API init in the shared - `include/onnxruntime/ep/api.h`. Do not try to force `ORT_API_MANUAL_INIT` on for everyone: MSVC's - `#pragma detect_mismatch` requires whole-binary agreement and doing so produced roughly 50 `LNK2038` errors. - -Files touched outside `docs/`: - -``` -.github/workflows/linux_webgpu.yml -.github/workflows/windows_webgpu.yml -cmake/CMakeLists.txt -cmake/onnxruntime.cmake -cmake/onnxruntime_providers_webgpu.cmake -cmake/onnxruntime_python.cmake -cmake/onnxruntime_unittests.cmake -include/onnxruntime/core/session/environment.h -include/onnxruntime/ep/adapter/op_kernel_info.h -include/onnxruntime/ep/api.h -onnxruntime/core/providers/webgpu/ep/api.cc -onnxruntime/core/providers/webgpu/math/{gemm,softmax,top_k}.h -onnxruntime/core/providers/webgpu/tensor/cast.h -onnxruntime/core/session/environment.cc -onnxruntime/core/session/ort_env.{h,cc} -onnxruntime/core/session/plugin_ep/ep_library_plugin.cc -onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.{h,cc} (new) -onnxruntime/core/session/plugin_ep/ep_library_static_plugin.{h,cc} (new) -onnxruntime/core/session/plugin_ep/ep_static_plugins.{h,cc} (new) -onnxruntime/test/framework/dynamic_plugin_ep_test.cc -onnxruntime/test/framework/inference_session_test.cc -onnxruntime/test/unittest_main/test_main.cc -onnxruntime/test/unittest_util/test_dynamic_plugin_ep.{h,cc} -tools/ci_build/{build.py,build_args.py} -``` - -## Building - -Three WebGPU configurations exist and it is worth being able to build at least two of them, because several -bugs in this work only appeared in one of them: - -``` -python tools/ci_build/build.py ... --use_webgpu # internal EP (the default, pre-existing path) -python tools/ci_build/build.py ... --use_webgpu shared_lib # plugin EP in its own shared library -python tools/ci_build/build.py ... --use_webgpu static_plugin # plugin EP linked into the host <-- the new one -``` - -`--use_webgpu static_plugin` maps to `-Donnxruntime_USE_EP_API_ADAPTERS=ON --Donnxruntime_WEBGPU_STATIC_PLUGIN=ON` (see `tools/ci_build/build.py`, the `--use_webgpu` handling). Both -options default to `OFF`. - -`onnxruntime_USE_EP_API_ADAPTERS=ON` is **incompatible** with `onnxruntime_BUILD_DAWN_SHARED_LIBRARY=ON` -(`cmake/CMakeLists.txt` raises a `FATAL_ERROR`). The internal-EP WebGPU CI job sets the latter; the plugin jobs -must not. - -### Windows gotchas - -These cost real time on the original machine and are likely to recur: - -- **`MAX_PATH`.** Building WebGPU under a long path such as `\build\Windows` makes Dawn's DXC build fail - with `MSB3491`. Build into a short directory instead — `C:\b` and `C:\b2` were used for the `static_plugin` - and `shared_lib` configurations respectively. With the Visual Studio generator the binaries land in - `\\\`. - - Refinement from a later machine: this is really an **MSBuild** problem. With `--cmake_generator Ninja` and - the registry's `LongPathsEnabled=1`, building at `D:\source\onnxruntime_4\build\sp` worked fine, so an - in-repo build directory is usable if you avoid MSBuild. Note that with Ninja the binaries land in - `\\`, one level, not two. -- **CMake may not be on `PATH`.** On a Visual Studio install it is at - `C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin`. -- Incremental rebuild of just the EP is much faster than a full build: - `cmake --build --config Debug --target onnxruntime_providers_webgpu --parallel`. - -### Verifying the D2 export contract - -For the **shared library** configuration, confirm the exported entry points are still unprefixed, since -`EpLibraryPlugin::Load` resolves them by name: - -``` -dumpbin /exports \\\onnxruntime_providers_webgpu.dll -``` - -Expect exactly `CreateEpFactories` and `ReleaseEpFactory`. The `static_plugin` build should instead produce -`WebGpu_`-prefixed symbols and no DLL. - -## Testing - -Relevant tests: - -- `onnxruntime_test_all --gtest_filter=InferenceSessionTests.WebGpu*` — covers registration, device - enumeration, and session creation end to end through the public V2 API. -- `onnxruntime_provider_test` — WebGPU EP kernel tests. - -The original development machine was a **Hyper-V VM with no real GPU**. On such a host WebGPU only surfaces a -*virtual* `OrtEpDevice`, and any session that is not compile-only is rejected up front with -*"WebGPU EP was selected on a virtual GPU device…"*. So locally only these two pass: - -- `InferenceSessionTests.WebGpuVirtualDeviceCompileOnlyEndToEnd` -- `InferenceSessionTests.WebGpuVirtualDeviceRejectedWithoutCompileOnly` - -and `WebGpuCompileOnlySkipsFinalization` and `TestStrictShapeInference` fail. This was proven to be a property -of the host and not of static linking, by building `--use_webgpu shared_lib` on the same machine and observing -the identical pass/fail set. - -**Consequence for whoever picks this up:** operator execution cannot be validated on a GPU-less machine at all. -Two ways to get real coverage: - -1. **Use CI.** The new `webgpu_static_plugin_build_x64_RelWithDebInfo` job in `.github/workflows/windows_webgpu.yml` - runs on the `onnxruntime-github-Win2022-GPU-A10` pool, which is the only GPU-equipped WebGPU pool, and it - both builds and tests. It has passed. This is currently the primary source of operator-level truth. -2. **Run locally on a real GPU.** Nothing in the change is GPU-vendor specific; a normal Windows machine with a - D3D12 or Vulkan capable GPU is enough. - -### Results on a real GPU - -This has now been done, on a machine with an NVIDIA RTX 5060 Ti. Both configurations were built from the same -tree, RelWithDebInfo, Ninja, differing **only** in the EP path: - -``` -python tools\ci_build\build.py --config RelWithDebInfo --build_dir --skip_submodule_sync --parallel \ - --cmake_generator Ninja --use_webgpu [static_plugin] --update --build --skip_tests \ - --cmake_extra_defines onnxruntime_BUILD_UNIT_TESTS=ON \ - onnxruntime_ENABLE_DAWN_BACKEND_D3D12=1 onnxruntime_ENABLE_DAWN_BACKEND_VULKAN=1 -``` - -`onnxruntime_provider_test`, with `ONNXRUNTIME_TEST_GPU_DEVICE_ID=0`: - -| | static_plugin | internal EP | -| --- | --- | --- | -| Total | 5894 | 5938 | -| Passed | 5775 | 5803 | -| Failed | 0 | 0 | -| Skipped | 119 | 135 | - -**Across the 5894 tests common to both builds there were zero status differences.** This is the evidence that -routing the static build through the plugin path changes no operator behaviour. - -`onnxruntime_test_all`: 1965 passed on both, no differences. All three `InferenceSessionTests.WebGpu*` pass on -the `static_plugin` build, including `WebGpuCompileOnlySkipsFinalization` — confirming the earlier failure of -that test was a property of the GPU-less host, as suspected. - -Two categories of test legitimately differ between the builds; neither is a regression: - -- **44 tests exist only in the internal-EP build.** 16 are `DISABLED_` and never run. The other 28 are - white-box tests that include internal WebGPU EP headers (`WebGpuContextTest`, `ActivationCacheKeyTest`, - `MatMul2BitsWebGpu`, `HardSwish_WebGPU`); `cmake/onnxruntime_unittests.cmake` (the - `onnxruntime_USE_WEBGPU AND NOT onnxruntime_USE_EP_API_ADAPTERS` guard) excludes `test/providers/webgpu/*` - from adapter builds entirely. Of these, `HardSwish_WebGPU` and `MatMul2BitsWebGpu` (12 tests) are real - operator coverage that exists only on the internal path — a known coverage gap of the plugin path, worth - either documenting in the design doc or closing by porting them to EP-agnostic tests. -- **The `InferenceSessionTests.WebGpu*` virtual-device tests are deliberately paired, not shared.** The - internal build has `WebGpuEpFactoryVirtualDevice`, `WebGpuEpFactoryRejectsVirtualDeviceWithoutCompileOnly` - and `WebGpuCompileOnlyUsesNoOpAllocator`; the plugin build has `WebGpuVirtualDeviceCompileOnlyEndToEnd` and - `WebGpuVirtualDeviceRejectedWithoutCompileOnly`, which `inference_session_test.cc` documents in-source as the - plugin-build counterparts. Only `WebGpuCompileOnlyUsesNoOpAllocator` has no counterpart, because it asserts - an internal allocator *type* via `dynamic_cast`, which is inherently unobservable across the ABI boundary. - -One caveat for whoever repeats this: do **not** run the two `onnxruntime_test_all` binaries concurrently. -`PathValidationTest`'s symlink tests collide over a shared temp directory and produce a spurious, and -non-deterministic, mix of pass/fail/skip. Run sequentially: all 31 pass on both builds. - -There is also a `webgpu-local-testing` skill in `.agents/skills/` describing how to run WebGPU tests on Linux -without a GPU using Mesa lavapipe (software Vulkan). Note its stated limitation: any graph containing `MatMul` -crashes lavapipe, so it only validates host-side enforcement and shape logic. - -### Not covered by any test yet - -Re-registration — register, run, unregister, re-register — is untested and is not reachable from the existing -test binaries, because `OrtEnv` is a refcounted process singleton that the unit test main pins for the lifetime -of the process. - -## CI - -No pre-existing leg built the `static_plugin` configuration, since both CMake options default to `OFF`. Every -WebGPU job was either internal EP (`--use_webgpu`) or shared library plugin (`--use_webgpu shared_lib`). Two -jobs were added: - -- `webgpu_static_plugin_build_x64_RelWithDebInfo` in `.github/workflows/windows_webgpu.yml` — builds **and - tests** on the A10 GPU pool. It deliberately omits `onnxruntime_BUILD_DAWN_SHARED_LIBRARY=ON` (incompatible, - see above) and deliberately drops the `--disable_rtti` / `--enable_lto` flags that the shared library job - carries, since those describe how the plugin ships as a DLL and here they would apply to all of ORT. -- `build-linux-webgpu-static-plugin-x64-release` in `.github/workflows/linux_webgpu.yml` — build only, mirroring - its sibling job. This is the only **GCC** coverage of the EP adapters, and the only coverage of the - `cmake/onnxruntime_python.cmake` change, which is why `--build_wheel` is kept on it. - -`linux_webgpu.yml` is build-only by design; its test job is commented out with a pre-existing "currently -failing" TODO. - -### Running CI on this branch - -The workflows only trigger automatically on `push`/`pull_request` for `main`, `rel-*`, and -`plugin-ep-webgpu/rel-*`, so **pushing this branch triggers nothing**. Dispatch manually: - -``` -gh workflow run windows_webgpu.yml --repo microsoft/onnxruntime --ref edgchen1/webgpu_static_plugin -gh workflow run linux_webgpu.yml --repo microsoft/onnxruntime --ref edgchen1/webgpu_static_plugin -gh workflow run windows_cuda_plugin.yml --repo microsoft/onnxruntime --ref edgchen1/webgpu_static_plugin -``` - -`gh workflow run --ref ` reads the workflow file **from that ref**, so jobs added on the branch do run -even though they are not on `main` yet. This was confirmed empirically. - -`windows_cuda_plugin.yml` matters because the CUDA plugin EP shares `include/onnxruntime/ep/*` with WebGPU, so -it is the regression check for the D5 and D7 changes. - -### Retrieving logs - -`gh run view --log-failed` and `gh run view --job --log` both returned **zero bytes** for these runs. Use -the REST API instead: - -``` -gh api repos/microsoft/onnxruntime/actions/runs//jobs --jq '.jobs[] | select(.conclusion=="failure") | "\(.id) \(.name)"' -gh api repos/microsoft/onnxruntime/actions/jobs//logs > log.txt -``` - -When grepping the result, search for `FAILED:` **case-sensitively**. A case-insensitive search for "failed" or -a search for `error:` drowns in benign CMake and Dawn feature-probe output such as -`-- Performing Test ... - Failed`. The same trap applies to local build logs: `grep -c "error:"` matches Dawn's -own configure line `-- DAWN Werror: OFF`, which looks alarming in a summary count and is nothing. - -### Run history - -| Run | Workflow | Result | -| --- | --- | --- | -| 32914301463 | ONNX Runtime WebGPU Builds | Success, all 6 jobs including the new one | -| 32914304408 | Linux WebGPU CI | Failure — GCC `maybe-uninitialized`, batch 1 | -| 32914307098 | CUDA Plugin Windows CI | Failure — `ParseInitializationConfigRejectsMissingRequiredFields` | -| 32996643911 | Linux WebGPU CI | Failure — GCC `maybe-uninitialized`, batch 2 | -| 32996646394 | CUDA Plugin Windows CI | Success | -| 33009901327 | Linux WebGPU CI | Failure — GCC `array-bounds`, batch 3. Internal-EP job on the same run passed. | -| 33030694040 | Linux WebGPU CI | **Success** — both jobs. Confirms the `array-bounds` fix; first green GCC run on this branch. | -| 33030695742 | ONNX Runtime WebGPU Builds | Dispatched on `ef68884a95` — re-run after the merge from `main` | -| 33030697208 | CUDA Plugin Windows CI | Dispatched on `ef68884a95` — re-run after the merge from `main` | - -## GCC and warnings as errors - -This was the only substantial problem the new CI surfaced, and it will bite again, so it is worth understanding -rather than pattern-matching. - -The EP adapter types are defined **entirely inline in headers**. The in-tree `OpKernelInfo` hides its -accessors behind out-of-line definitions in a `.cc`, so GCC cannot see their failure and exception paths. With -the adapters, it can — and it then warns about values that are only ever read once the call has succeeded. The -warning is a false positive, but the build uses `-Werror`. - -This had never been observed before simply because the EP-adapters path had never been compiled with GCC: the -existing Linux WebGPU CI builds the internal EP, and the shared library plugin builds are Windows/MSVC-only. -The CUDA plugin EP hit the identical problem earlier and solved it with a target-scoped suppression in -`cmake/onnxruntime_providers_cuda_plugin.cmake`. - -Two batches of failures appeared, because ninja stops early and only reports the targets that were already in -flight — **do not assume the first batch is the complete list**. A third batch, of a different warning class, -appeared later still. The lesson is the general one: **each CI round trip reveals only one batch**, so prefer a -local GCC build (or a throwaway dispatch with `--compile_no_warning_as_error`) to enumerate all of them at once. - -- Batch 1, fixed at the source in `49855f6ef7`: uninitialized locals that are passed by pointer and read back, - and the temporaries inside `GetAttrOrDefault` / `GetAttr` in `include/onnxruntime/ep/adapter/op_kernel_info.h`. - Value-initializing them (`T tmp{}`) is correct and worth keeping on its own merits. -- Batch 2: false positives deep inside `absl::InlinedVector`, for `TensorShapeVector` locals in - `tensor/pad.cc` and `tensor/unsqueeze.h` that are always properly constructed. These cannot be fixed at the - source. Handled by applying the same `-Wno-maybe-uninitialized` suppression the CUDA plugin EP uses, scoped - to the `onnxruntime_providers_webgpu` target, GCC only, and only when `onnxruntime_USE_EP_API_ADAPTERS` is on - so that non-adapter builds stay strict. -- Batch 3, in run `33009901327`, **fixed at the source**: `-Werror=array-bounds`, a *different* warning class, at - `onnxruntime/contrib_ops/webgpu/quantization/matmul_nbits_mlp.cc:47` in `EmitGateActivationExpr`: - - ``` - ‘void* __builtin_memcpy(...)’ forming offset [32, 48] is out of the bounds [0, 32] - of object ‘’ with type ‘std::__cxx11::basic_string’ - ``` - - GCC 14 at `-O3`, inlining the `std::string{gate_var} + " * (one / (one + exp(-" + std::string{gate_var} + ")))"` - temporary-move concat chain through `std::operator+(basic_string&&, const char*)`. A false positive. Note that - the batch-2 `-Wno-maybe-uninitialized` suppression **is** working — it is present on the failing command line - and the batch-2 files compiled clean. The file is pre-existing (added by PR #28280, `88ca23fcd5`), not authored - by this branch; the adapters PCH pulls in `include/onnxruntime/ep/adapters.h`, which changes inlining and is - why only the adapters config trips it. Fixed by building the expression with ORT's `MakeString` instead of a - `std::string` concat chain — a single-site source fix, preferred over extending the target-scoped suppression - with `-Wno-array-bounds`, which would have blinded the whole provider to a genuinely useful warning. - -### The complete GCC 14 warning inventory - -The batch-at-a-time discovery loop was finally closed by enumerating **every** warning in one pass: a full -container build with `--compile_no_warning_as_error`, then grouping the log by warning class. The result: - -| Warning class | Count | Where | -| --- | --- | --- | -| `-Wattributes` | 276 | Dawn generated code (`dawn_proc.cpp`) | -| `-Wunused-parameter` | 55 | ONNX (`onnx/defs/**`) | -| `-Wdangling-pointer=` | 3 | Dawn / Tint | -| `-Wchanges-meaning` | 3 | Dawn (`SystemEvent.h`) | -| `-Wstringop-overflow=` | 2 | flatbuffers (`reflection.cpp`) | -| `-Warray-bounds=` | **1** | **ORT — `matmul_nbits_mlp.cc:47`** | -| `-Wstringop-overread` | 1 | telemetry (`sqlite3_retail.c`) | - -**Exactly one warning in the whole build is attributable to ORT's own source**, and it is the batch-3 failure. -Everything else is third-party, in targets that do not compile with `-Werror`. So there is no batch 4: fixing -this one site is expected to turn the leg green. - -### Reproducing the CI compiler locally - -Worth the setup cost — it turns a ~28 minute CI round trip per batch into a local loop, and it is the only way -to enumerate all batches at once. - -The CI base image (`onnxruntimebuildcache.azurecr.io/.../cpu_x64_almalinux8_gcc14`) requires ACR auth and is -**not pullable**. An equivalent image is easy to build, and reproduces the compiler exactly -(GCC 14.2.1 20250110, Red Hat) — `almalinux:8` + `epel-release` + `gcc-toolset-14` + `python3.11`, with -`cmake ninja packaging numpy setuptools wheel` from pip. Notes: - -- **Python 3.11 is required, not the distro's 3.9** — `tools/ci_build/build_args.py` uses a `match` statement - (3.10+). CI itself uses `/opt/python/cp310-cp310/bin`. -- `--allow_running_as_root` is required inside the container. -- **No X11 dev packages are needed**, because `cmake/external/onnxruntime_external_deps.cmake` sets - `DAWN_USE_X11 OFF` unconditionally. This is what keeps the image minimal. -- Clone into the WSL native filesystem, not `/mnt/d` — building over the 9p mount is crippling. Only - `cmake/external/onnx` and `cmake/external/libprotobuf-mutator` need initialising for a native Linux build. - -To re-check a single file without paying for a whole build, override `CXX_FLAGS` on the generated makefile -rule, which lets you promote warnings back to errors against an existing build tree: - -```bash -FLAGS=$(sed -n 's/^CXX_FLAGS = //p' CMakeFiles/onnxruntime_providers_webgpu.dir/flags.make) -make -f CMakeFiles/onnxruntime_providers_webgpu.dir/build.make \ - CMakeFiles/onnxruntime_providers_webgpu.dir/src//matmul_nbits_mlp.cc.o \ - CXX_FLAGS="$FLAGS -Werror" -``` - -Always run the **negative control** — rebuild the unfixed source through the same harness and confirm it still -errors. Otherwise a harness that silently fails to exercise the warning looks exactly like a successful fix. - -### Verification of the fix - -Both halves were checked, because each one alone is inconclusive: - -1. **Single-object recheck.** With the fix, `matmul_nbits_mlp.cc.o` compiles clean under `-Werror`. Reverting the - file and rebuilding through the identical harness reproduces the exact CI error, which is what proves the - harness was actually exercising the warning. -2. **Full build with `-Werror` on**, i.e. the real CI configuration with `--compile_no_warning_as_error` dropped: - completes to 100%, links `onnxruntime_provider_test`, zero compiler errors. Before trusting this, confirm - `-Werror` really is on the ORT targets — `sed -n 's/^CXX_FLAGS = //p' CMakeFiles/.dir/flags.make` - should show it for `onnxruntime_providers_webgpu`, `onnxruntime_providers` and `onnxruntime_session` — and - confirm the fixed file appears in the build log, so it was recompiled rather than served from a stale object. - -The 334 warnings that remain in that build are all third-party, in targets that do not use `-Werror`. - -## Emscripten and ORT Web - -This is the motivating use case for the whole change, so it was built locally before opening the PR. **It builds -and links, and the plugin registration is verifiably present in the output.** - -Note what ORT Web's own CI does *not* cover today: the WebGPU WASM jobs in -`.github/workflows/linux-wasm-ci-build-and-test-workflow.yml` pass a bare `--use_webgpu`, and bare `--use_webgpu` -defaults to `static_lib` (`tools/ci_build/build_args.py`, `const="static_lib"`). So every existing WASM job still -builds the **internal** EP. Nothing in CI exercises WASM + `static_plugin`; this branch only makes the -configuration available, it does not switch ORT Web over to it. - -The CMake side was already written with Emscripten in mind — `cmake/onnxruntime_providers_webgpu.cmake` rejects -`shared_lib` there with `"WebGPU EP shared library build is not supported on Emscripten. Please use -'--use_webgpu static_plugin'."` - -### Building it - -Same container as the GCC 14 reproduction (see [above](#reproducing-the-ci-compiler-locally)), plus: - -- `git submodule update --init --recursive cmake/external/emsdk` — not needed for a native Linux build, required - here. `build.py` installs and activates the toolchain itself. -- **`node` must be on `PATH`.** `cmake/external/onnxruntime_webassembly.cmake` includes `node_helper.cmake`, - which does a hard `find_program(NODE_EXECUTABLE ...)` and fails configure with - `Could not find NODE_EXECUTABLE using the following names: node.exe, node`. CI gets node from - `actions/setup-node`; the AlmaLinux image has none. emsdk bundles one — point `PATH` at - `cmake/external/emsdk/node//bin`. - -``` -docker run --rm -v /home/edch/ort:/src -v /home/edch/ortwasm:/build -w /src ort-gcc14:local \ - bash -lc "export PATH=/src/cmake/external/emsdk/node/22.16.0_64bit/bin:\$PATH && \ - python3.11 tools/ci_build/build.py --build_dir /build --config Release --skip_submodule_sync --parallel \ - --allow_running_as_root --build_wasm --enable_wasm_simd --enable_wasm_threads \ - --enable_wasm_api_exception_catching --use_webgpu static_plugin --use_webnn \ - --target onnxruntime_webassembly --skip_tests" -``` - -Result: configure clean, zero compile failures, links `ort-wasm-simd-threaded.asyncify.mjs` + -`.wasm` (~32 MB). Roughly 35 minutes including the emsdk toolchain download and sysroot generation. - -### Verifying the plugin path actually linked - -A successful build alone proves little here, because the registration is behind -`#if defined(USE_WEBGPU) && defined(ORT_WEBGPU_STATIC_PLUGIN)` in -`onnxruntime/core/session/plugin_ep/ep_static_plugins.cc`. If that define failed to reach ORT core, the file -would still compile — `CreateStaticPluginEpLibraries()` would just return an empty vector, and the build would -be just as green while registering nothing. - -The decisive check is that ORT core holds an **undefined** reference to the prefixed entry points, using the -emsdk LLVM tools: - -``` -NM=cmake/external/emsdk/upstream/bin/llvm-nm -$NM /Release/libonnxruntime_session.a | grep WebGpu_ # expect: U WebGpu_CreateEpFactories, U WebGpu_ReleaseEpFactory -$NM /Release/libonnxruntime_providers_webgpu.a | grep WebGpu_ # expect: T WebGpu_CreateEpFactories, T WebGpu_ReleaseEpFactory -``` - -Observed exactly that: `U` in core, `T` in the provider, and the link resolved them. That closes the loop — -the `#if` was live, the entry point prefixing worked, and `EpLibraryStaticPlugin` is in the binary. - -### Running it in a browser - -The wasm binary has now been **run**, on a real GPU, in Edge. A dedicated harness lives in `build/bench_web/` -(untracked scratch, not part of the repo): - -- `make_side.ps1 -Side ` builds one **self-consistent "side"**: it copies that side's - `ort-wasm-simd-threaded.asyncify.{wasm,mjs}` into `js/web/dist`, checks out the matching - `js/web/lib/wasm/session-options.ts`, runs `npm run build -- --bundle-mode=perf --webgpu-ep`, and snapshots the - result with a `MANIFEST.txt` recording git SHA, source provenance and SHA-256 of every artifact. -- `serve.js` serves the snapshots on port 8099; `bench.js` is the page. -- `run_bench.ps1 -Sides @(...)` drives an interleaved (ABBA / cyclic-rotation) run across N sides. - -Two properties of the harness matter more than the timings: - -- **`--webgpu-ep` is mandatory and fails silently if omitted.** Without it `js/web/script/build.ts` sets - `DISABLE_WEBGPU=true` and the session quietly runs on **JSEP** instead, producing a plausible-looking but - completely irrelevant number. -- **A bundle and a wasm are a matched pair.** `build.ts` has an esbuild `onLoad` hook that *inlines the `.mjs` - loader into the bundle*, so a bundle from one build silently carries another build's loader. Sides must never - be mixed by hand. - -### Verifying the plugin path is actually taken at runtime - -Linking is not execution, and the failure mode here is silent, so the runtime path was pinned down with -controls rather than by reading code: - -| control | bundle | wasm | result | -| --- | --- | --- | --- | -| `plugin` | plugin | `static_plugin` | runs, correct output | -| `mismatch` | plugin | `static_lib` | **runs, correct output** — the silent trap | -| `mismatch2` | lib | `static_plugin` | fails: "WebGPU execution provider is not supported in this build" | -| bogus EP name | patched | `static_plugin` | fails: "No execution provider device is registered for `…BOGUS`" | - -The bogus-name failure goes on to list the registered devices — `CPUExecutionProvider` and -`WebGpuExecutionProvider`. That control is the load-bearing one: it proves the V2 name lookup is real and -enforced, so a *successful* `plugin` run necessarily resolved a genuinely registered plugin EP device. - -The page also patches `GPUQueue.prototype.submit` to count GPU submissions. That counter has been **perfectly -deterministic** across every run, which makes it a far better endpoint than any timing statistic. - -### Measured cost of the plugin path on the web build - -Comparing `static_lib` against `static_plugin` wasm built from the *same* commit — the two `build.py` argument -dumps are identical except `use_webgpu='static_lib'` vs `'static_plugin'` — over 6-8 interleaved rounds per -model, taking the median of each round's minimum: - -| model | nodes | `static_lib` | `static_plugin` | delta | predicted @ 4.76 us/node | residual | -| --- | --- | --- | --- | --- | --- | --- | -| `bench_compute` | 16 | 5.00 ms | 5.00 ms | 0.00 ms | 0.08 ms | -0.08 ms | -| `bench_dispatch_150` | 150 | 4.30 ms | 5.10 ms | +0.80 ms | 0.71 ms | +0.09 ms | -| `bench_dispatch` | 300 | 7.90 ms | 9.40 ms | +1.50 ms | 1.43 ms | +0.07 ms | -| `bench_dispatch_600` | 600 | 13.80 ms | 16.60 ms | +2.80 ms | 2.86 ms | -0.06 ms | - -The plugin side was slower in **26 of 26** paired rounds. A single through-origin constant of **~4.8 us of CPU -per kernel node** fits all four points to within one 0.1 ms timer tick, and **GPU submit counts were -byte-identical on every side, model and round** (3 per run for compute; 11/20/39 for 150/300/600). So this is -pure per-kernel CPU dispatch overhead, not extra queue work. The 16-node null is not evidence of no cost — the -predicted 0.08 ms simply sits below the timer floor. - -A three-way run isolated where the cost is **not**: the `mismatch` side (plugin bundle, `static_lib` wasm) came -out level with `lib` (median delta -0.20 ms, a 3/6 sign split), while `plugin` was +1.60 ms in 6/6 rounds. -The JS-side `_OrtAppendExecutionProviderV2` call and EP-name lookup are therefore free; the cost is inside the -`static_plugin` wasm's per-kernel C++ path. Where exactly is still unknown — 4.8 us is roughly 10k cycles, -far too much for an indirect call, so it points at real per-kernel work. An `--enable_wasm_profiling` build -would give named frames. - -Timer resolution is 0.1 ms: Edge coarsens `performance.now()` without cross-origin isolation. Enabling -COOP/COEP would give ~5 us resolution and was deliberately *not* turned on mid-experiment, to keep all numbers -comparable. - -### Correction: what the shared-allocator commit does and does not explain - -An earlier reading of these results as "refuting" the design doc's +36% figure was **wrong**, and the mistake is -worth recording. Commit `b876d290cd` ("Defer zero-initialize submission in the WebGPU plugin EP shared -allocator") is an **ancestor** of the benchmarked branch head, so every `plugin` number above is already -*plugin-with-the-fix*. Parity on the compute-bound model with identical submit counts is therefore -**consistent with that fix working**, not evidence against the original measurement. - -A second retraction: the claim that the earlier *native* A/B was "null by construction" because -`cmake/onnxruntime_providers_webgpu.cmake` excludes the `ep/` folder from non-plugin builds was also wrong. The -build logs show both native sides were `use_webgpu='shared_lib'`, which **does** compile `ep/factory.cc`. That -null was genuine. - -What remains genuinely untested is whether the deferral matters on **web/wasm** at all. That was settled by -a four-side experiment — `main_lib` (merge-base `2fee06a0c1`), `lib`, `plugin_nofix` (branch head with -`b876d290cd` reverted, commit `a6be215b31`) and `plugin` — each built from its **own worktree and its own build -directory**, with no reuse and no `--skip_submodule_sync`. The **primary endpoint is the deterministic submit -count, not timing**: the commit's own mechanism claim is "a queue submit for every intermediate buffer allocated -mid-run", so `plugin_nofix` must show more submits per run than `plugin` if it does anything at all. -`bench_dispatch` has 299 intermediates against `bench_compute`'s 15, making it ~20x the detector. - -### Result: `b876d290cd` has no measurable effect on the web build - -The submit counter is perfectly deterministic — the same value in all 8 rounds of every side, both models: - -| model | dispatches/run | submits/run: `main_lib` | `lib` | `plugin_nofix` | `plugin` | -| --- | --- | --- | --- | --- | --- | -| `bench_compute` | 16 | 3 | 3 | 3 | 3 | -| `bench_dispatch` | 300 | 20 | 20 | 20 | 20 | - -**Reverting the commit changes nothing.** On `bench_dispatch`, 300 dispatches with 299 intermediates still batch -into exactly 20 submits whether the deferral is present or not. The mechanism the commit describes — one submit -per intermediate buffer — simply **does not occur on the wasm build**, so there is nothing there for it to defer. - -Timing agrees, and is noise: `plugin - plugin_nofix` med `dmin` is **+1.70 ms (6/8 rounds *slower*)** on -`bench_dispatch` but **-0.30 ms (3/8)** on `bench_compute`. The sign flips between models, which is the -signature of build-to-build variation (code layout / icache) rather than a real effect. Note the direction: the -side *with* the fix is if anything marginally slower on the dispatch-heavy model. - -Combined with the earlier genuine native A/B null, `b876d290cd` has **no demonstrated benefit on any platform -tested**, and no correctness change was identified. It should not be landed on its own merits. - -Two other contrasts fall out of the same run, on the `medmin` delta: - -| pair | `bench_compute` | `bench_dispatch` | reading | -| --- | --- | --- | --- | -| `lib - main_lib` | 0.00 ms (0.0%) | -0.10 ms (-0.8%) | the branch does **not** regress the static_lib path | -| `plugin - lib` | +0.30 ms (+5.8%) | +3.00 ms (+25.2%) | cost of the plugin path itself | -| `plugin - main_lib` | +0.30 ms (+5.8%) | +2.90 ms (+24.2%) | **total ship delta vs. `main`** | - -The `lib` vs `main_lib` null is the important control: it isolates the plugin-path cost from "anything else the -branch changed". The isolation is unusually clean here because the two sides' `ort.all.min.js` came out -**byte-identical** (SHA-256 `79391F71...`), the merge-base and `main` blobs of `session-options.ts` being the -same — so the *only* difference between those two sides is the `.wasm`. The same holds for -`plugin_nofix` vs `plugin` (bundle `AB2F004F...`). All four `.wasm` hashes were confirmed distinct. - -### Disposition: `b876d290cd` reverted - -Given no measurable benefit on either the web or the native build, and no correctness issue that it fixes, the -commit was reverted rather than shipped. The revert restores the unconditional submit — the conservative -behaviour — so what is lost is a speculative optimization, not a guarantee. - -The revert was applied as a cherry-pick (a draft PR is already open, so history was not rewritten) and then -verified rather than assumed: - -- `git diff b876d290cd^ HEAD` over the five affected files is **empty**, and `git diff b876d290cd HEAD` is - **non-empty** — the tree is byte-identical to the pre-fix state. -- No dangling references to the removed `WebGpuContext` members remain (`IncrementActiveRunCount`, - `DecrementActiveRunCount`, `active_run_count_` — zero hits tree-wide). This matters because a clean - cherry-pick only proves the patch *applied*, not that later commits had not started using the deleted API. -- All three reverted `.cc` files recompile: `ep/factory.cc` and `webgpu_execution_provider.cc` in a - `static_plugin` build, and `webgpu_context_test.cc` in an integrated build. Note that no single config - compiles all three — `ep/factory.cc` is absent from the integrated build and the test is absent from the - plugin build — so both configs are needed for full coverage. - -The EP-level `WebGpuExecutionProvider::IsRunActive()` predates this commit and is unaffected; only the -context-level copy of the state was removed. - -The design doc's write-up of this detour — the `+36%` figure and the section proposing the deferred-submission -fix — has been removed from `static_plugin_ep_registration_design.md`, since that document describes the design -as it stands rather than the path taken to it. This work log keeps the record. - -Caveat on scope: this is "no measurable benefit in the scenarios we could construct", not proof of a universal -no-op. The evidence is strong for the web build, where the patched `GPUQueue.submit` counter is a *complete* -instrument for the claimed harm. It is weaker for native, where only timing was compared and submits were -never counted. - -## Native shared-library plugin measurements - -The latency criterion covers two shapes of plugin, and the wasm numbers above only settle the static one. The -native `--use_webgpu shared_lib` A/B against a built-in baseline is written up separately in -[native_plugin_latency_measurements.md](native_plugin_latency_measurements.md), with the harness in -`run_native_ab.ps1`. Summary: the plugin is slower by ~8% on `bench_dispatch` and 4-6% on Qwen3.5-0.8B, fitting -roughly **67 us per `Run` plus 1.2 us per node** — about 3-4x cheaper per node than the wasm boundary's 4.8 us. - -Three things from that exercise are worth recording here, because each cost real time and none was visible from -reading code: - -- **The first run measured the wrong GPU.** A driver failure had left the RTX 5060 Ti in PnP error state, so - every WebGPU run silently landed on a Quadro P620, which lacks `shader-f16` and could not run the int4 Qwen - model at all. Nothing in ORT's output names the adapter, and there is no adapter-selection knob, so on a - multi-GPU box the adapter has to be confirmed externally with `nvidia-smi` during a run. -- **One arm was accidentally built unoptimized.** An interrupted CMake configure left `CMAKE_CXX_FLAGS_RELEASE` - empty; a later `build.py --update` reused the damaged cache instead of regenerating it, and the build - succeeded with no error. Counting `/O2` in the generated `build.ninja` catches this; the cache alone does not - make it obvious. Had it gone unnoticed it would have produced a large bogus result favouring the plugin. -- **A CPU-EP control was needed to attribute the delta.** Matching build flags do not prove two executables - perform alike. Running both binaries on the CPU EP, where no plugin is loaded, gave a ratio of minima of - 1.001 — which is what licenses reading the WebGPU delta as plugin-boundary cost. Run with default threading - that same control was misleading, showing the plugin executable 24% *faster*; it only became usable - single-threaded. - -The relationship to the wasm result is the interesting part: both boundaries charge per kernel node, and the -native one charges less. That is consistent with the wasm cost being dominated by something inside the -`static_plugin` wasm's per-kernel C++ path rather than by the plugin indirection itself, which is what the -three-way `mismatch` run above already suggested. - -### Root cause of the native per-node cost - -1.2 us/node is ~4,400 cycles, which is far more than a cross-DLL indirect call can account for, so both arms were -CPU-profiled to find out what it is actually spent on. The write-up is in -[native_plugin_latency_measurements.md](native_plugin_latency_measurements.md#where-the-per-node-cost-goes-cpu-profile). -Outcome: the overhead is **not** boundary-crossing cost. It is redundant per-kernel work that the built-in path does -not do — `ep::adapter::CreateTensorFromApiValue` rebuilds an `onnxruntime::Tensor` (nine C API round trips, an -allocator-name `std::string`, a `TensorShape` allocation) for every input and output of every node on every `Run`, -because the adapter's `OpKernelContext` is constructed fresh per `Compute()` and caches nothing beyond it. The -adapter symbols are absent from the built-in profile entirely, `Tensor`/`TensorShape` work doubles, and the total -extra CPU (+0.53 ms/rep) accounts for the entire extra wall latency (+0.42 ms/rep). - -Three method notes, since each cost time: - -- **ORT's own `-p` profiler cannot do this A/B.** The built-in arm emits 15,300 `Api` category events and the plugin - arm emits zero, so under `-p` the built-in arm measures *slower* and the result inverts. That asymmetry is also a - finding in its own right: the plugin path silently loses the WebGPU EP's `Api` profiling events, an observability - regression worth fixing independently of latency. -- **ETW/xperf symbolization does not work on this machine.** Traces collect fine but every frame renders as - `***unknown***` for every module, with symcache generated and with or without the Microsoft symbol server. Do not - spend time on it. -- **What worked was a purpose-built sampling profiler** (`D:\test\prof\sampler.cpp`, outside the repo): sample every - thread's instruction pointer at ~1 kHz, symbolize with dbghelp against local PDBs after exit. Needs no elevation. - The binaries were relinked with `/DEBUG` only — `LINK_FLAGS` already set `/OPT:REF,ICF,LBR` explicitly, so no - codegen changed and the A/B ratio was re-verified afterwards. - -## PR review feedback - -Draft PR [#32395](https://github.com/microsoft/onnxruntime/pull/32395) collected eleven inline self-review -comments. All eleven are addressed. Two turned out to rest on a premise the code does not support, and are -recorded here because the resolution differs from the comment. - -**`OrtAppendExecutionProviderV2` should verify there is only one matching device.** It should not: erroring out -on more than one match would be wrong. `Factory::GetSupportedDevicesImpl` loops over every GPU hardware device, -and the virtual-device path adds its device *in addition to* any real ones, so several `OrtEpDevice`s -legitimately share one EP name. The comment was right that a single device has to be chosen, but the choice -belongs to the caller, not to the wasm wrapper — and the real defect was that the pick was silent and -undocumented rather than that it happened at all. - -The fix is bigger than the comment asked for, because a companion comment objected that the wrapper's name -claims a kinship with `SessionOptionsAppendExecutionProvider_V2` that its signature did not honour. Both are -settled by moving EP-device selection out of C++ and into JavaScript, so that the wasm export really is the -underlying API: - -- `OrtAppendExecutionProviderV2` now takes an `OrtEpDevice*` array and a count, exactly like the C API, and is - a straight pass-through. The name is kept, because now the signature does match. -- Two new exports supply what JavaScript needs to build that array: `OrtGetEpDevices`, a thin wrapper over - `GetEpDevices`, and `OrtEpDevice_EpName`, a 1:1 mirror of `EpDevice_EpName`. -- `session-options.ts` gains `getEpDevicesByEpName()`, which walks the returned array once into a - `Map`. Selection policy lives there, and the not-found error can list the registered names - because the map has them. - -The selection policy is to pass the **first** matching device. That is necessary because the WebGPU EP backs a -session with a single Dawn device, so `Factory::CreateEpImpl` rejects any device count other than 1 — passing -every match would break the moment a second device appeared. It is safe because the factory reads nothing from -the device except whether it is virtual: which GPU gets used is decided by the adapter ORT Web hands to Dawn, -not by the `OrtEpDevice`. Emscripten device discovery reports a single GPU entry derived from `navigator.gpu` -(there is no per-GPU enumeration in a browser, and `requestAdapter` takes only `powerPreference` / -`forceFallbackAdapter` hints), so in practice there is exactly one real candidate. A second entry appears only -when virtual devices are enabled, and the factory registers those after the real ones, which `Environment` -preserves. Even if that order changed, EP creation would fail with the explicit "selected on a virtual GPU -device" error rather than silently running on the wrong device. - -Selecting an `OrtEpDevice` from JavaScript was considered and is not worth building. It would need a public -enumeration API in `onnxruntime-common` plus a set of new wasm accessors for the device descriptor, and it -would still not choose a GPU, because the browser does not expose one to choose. Applications that need to -control the device already can: `env.webgpu.adapter` and the WebGPU EP's `device` option accept a -caller-created `GPUAdapter`/`GPUDevice`, which is both the browser-sanctioned mechanism and more expressive -than picking an `OrtEpDevice`. - -Filtering on the C++ side was considered and rejected. `GetEpDevices` returns an array *owned by the -environment*; passing the pointer straight through keeps it borrowed. Any C++-side filter would have to -materialise a new array and invent an ownership contract that this layer does not otherwise have. The selected -subset has to be marshalled into wasm memory by the caller either way, and `setExecutionProviders` already has -an `allocs` list that does exactly this for the EP option keys and values. - -Two traps in the JavaScript half, neither of which the type checker catches: - -- Pointers read out of wasm memory with `getValue(ptr, '*')` must be wrapped in `Number(...)`. Under `ORT_WASM64` - that call reads `HEAP64` and yields a **BigInt**, and `BigInt + Number` pointer arithmetic throws a - `TypeError`. Values *returned from* an exported function need no wrapping — Emscripten converts those at the - call boundary. `jsep/init.ts` and `wasm-core-impl.ts` follow the same split. -- Every new `_Ort*` export must be added to the `no-underscore-dangle` allow-list in `js/eslint.config.mjs`, or - `npm run lint` fails. The earlier commit that introduced `_OrtAppendExecutionProviderV2` missed this and left - the branch red; that is fixed here too. - -The device-array path was then run, not just compiled. `build\web_plugin` was relinked and a fresh `plugin` side -built, and two facts were checked against the pre-rework `plugin` results: - -| endpoint | before | after | -| --- | --- | --- | -| `bench_dispatch` output sample | `-0.547067046165466, 0.458441495895386, 0.369816303253174, 0.281192660331726` | identical | -| GPU submits per run | 20 | 20 | - -Submit counts are deterministic, so an identical count plus bit-identical outputs is a much stronger check than -any timing. The negative control is the load-bearing one: patching the bundle's EP name to -`BogusExecutionProviderX` now fails in JavaScript with - -``` -no execution provider device is registered for: BogusExecutionProviderX. -registered: CPUExecutionProvider, WebGpuExecutionProvider. -``` - -That message can only be produced by walking a real device array, so it proves `OrtGetEpDevices`, -the `Number(getValue(...))` pointer arithmetic, `OrtEpDevice_EpName` and the grouping all work — the parts a -successful run would otherwise exercise silently. Confirming that the two new exports are also *not* -asyncify-wrapped: `_OrtAppendExecutionProviderV2` appears twice in the generated `.mjs` (export plus JSPI -wrapper) while `_OrtGetEpDevices` and `_OrtEpDevice_EpName` appear once each. - -Narrowing the selection to a single device was re-verified the same way, against the same endpoint: the output -sample stayed bit-identical and `submitsPerRun` stayed a constant 20, with `300 + 61 x 20 = 1520` total submits -over the 10 warmup, 1 first and 50 steady runs. This machine registers one WebGPU device, so the change is -behaviour-preserving here by construction; what the run establishes is that the rewritten append block still -selects the WebGPU EP and produces identical results, not that the multi-device case is exercised. That case -needs virtual devices enabled, which ORT Web has no way to turn on today. - -**The Emscripten device discovery should detect a GPU in Node.js some other way.** There is no better way, and -the existing check is not as blind as the comment assumed. Node has never natively exposed `navigator.gpu`; the -request was closed as not planned ([nodejs/node#42896](https://github.com/nodejs/node/issues/42896)) and there -is no `--experimental-webgpu` flag in core. WebGPU in Node comes from userland Dawn bindings, which assign onto -`globalThis`, so the existing `EM_ASM` check already picks them up. A real enumeration would need the -asynchronous `requestAdapter()`, which cannot be called from discovery. Only the comment was wrong — it claimed -Node always reports no GPU — so this is a comment-only change. The file is under `core/platform/emscripten/` -and so never affects native `onnxruntime-node`, which has its own discovery. - -Two of the remaining nine were real defects rather than cleanups: - -- `OpKernelInfo::GetAttrsImpl` for `std::vector` leaked the allocator's array, and every string - still in it, if `reserve()` or a `std::string` copy threw. It now holds the array in a `unique_ptr` whose - deleter frees each element. The deleter calls `OrtAllocator::Free` through the function pointer rather than - `Ort::AllocatorWithDefaultOptions::Free()`, which throws on failure and so must not run in a destructor. This - only matters for native plugin EPs; on web, `-sDISABLE_EXCEPTION_CATCHING` turns `bad_alloc` into an abort. -- `OrtEnv::GetOrCreateInstance` unpublished and deleted the instance by hand when static plugin EP registration - failed, but a *throw* from that registration skipped the cleanup and left a published, half-built - `p_instance_` for the next caller to pick up. The instance is now held in an `OrtEnvPtr` taken immediately - after `++ref_count_`, which removes the manual `delete` and `ref_count_ = 0` as well. This is safe because - `OrtEnv::Release` re-enters `m_`, which is a `std::recursive_mutex` already held by this thread. - -The rest were naming and comment cleanups. The most substantive: the build defined three macros that were -perfectly correlated — `ORT_PLUGIN_EP_ENTRY_POINT_PREFIX` and `ORT_PLUGIN_EP_STATICALLY_LINKED` on the static -branch, `ORT_PLUGIN_EP_OWNS_PROCESS_GLOBALS` on the shared one. The first and third are gone; everything keys -off `ORT_PLUGIN_EP_STATICALLY_LINKED`, and the `WebGpu_` prefix is hardcoded. Parameterising only the provider -side was fictional flexibility anyway, since ORT core already spells `WebGpu_CreateEpFactories` out literally -in `ep_static_plugins.cc`. - -Separately, the manual-init switch in the public header `include/onnxruntime/ep/api.h` was renamed from -`ORT_PLUGIN_EP_STATICALLY_LINKED` to `ORT_PLUGIN_EP_SKIP_API_MANUAL_INIT`, since it says nothing about linkage — it says -whether `OrtGetApiBase()` is reachable in-process. The negative polarity is deliberate and was kept: undefined -must continue to mean manual init *on*, which is the safe default for a separately built plugin. A positive -spelling would silently flip that for any out-of-tree plugin that relies on the header. `webgpu/ep/api.cc` has -its own copy of the `ORT_API_MANUAL_INIT` dance and is keyed off the same macro, because MSVC's -`detect_mismatch` pragma requires every translation unit in a binary to agree. - -Verification: compiled in all three affected configurations — `build\int` for the core session objects, `build\sp` -for the whole `onnxruntime_providers_webgpu` target (which force-includes the EP adapter headers, so it is what -covers `op_kernel_info.h`), and `build\wasm_sp` for `wasm/api.cc` and the Emscripten `device_discovery.cc`. On the -RTX 5060 Ti machine, `onnxruntime_test_all --gtest_filter=InferenceSessionTests.WebGpu*` passes all three, and -`onnxruntime_provider_test` passes with no failures. - -## Open items - -1. ~~**Fix the Linux leg.**~~ **Done.** The batch-3 `array-bounds` break is fixed at the source, and a local - GCC 14 build confirmed it was the *only* ORT-owned warning in the whole build (see - [The complete GCC 14 warning inventory](#the-complete-gcc-14-warning-inventory)). CI run `33030694040` - is green on both jobs. -2. ~~**Operator-level validation on a real GPU.**~~ **Done** — see [Results on a real GPU](#results-on-a-real-gpu). - No behavioural difference. Remaining sub-item: decide whether to close the 12-test - `HardSwish_WebGPU` / `MatMul2BitsWebGpu` coverage gap on the plugin path, or document it as known. -3. ~~**Minimal build.**~~ **Resolved.** Exactly one pipeline combines an extended-minimal build with WebGPU — - `webgpu_minimal_build_edge_build_x64_RelWithDebInfo` in `.github/workflows/windows_webgpu.yml`, which passes a - bare `--use_webgpu` (internal EP) and is green, so the `#if !defined(ORT_MINIMAL_BUILD)` guard in `ort_env.cc` - is already exercised in CI. No ORT Web or wasm job uses a minimal build at all. The remaining hole — minimal - plus `static_plugin` linking the EP in without registering it — is now a configure-time `FATAL_ERROR` in - `cmake/CMakeLists.txt`, verified to fire on that combination and not on a non-minimal `static_plugin` - configure. Lifting the restriction properly is follow-up work, recorded in D4 of the design doc. -4. ~~**Emscripten / ORT Web.**~~ **Builds, links, and now runs in a browser** on a real GPU — see - [Running it in a browser](#running-it-in-a-browser). Registration is verified in the binary *and* at runtime. - Open sub-item: the plugin path costs **~4.8 us of CPU per kernel node** on the web build - ([measurement](#measured-cost-of-the-plugin-path-on-the-web-build)); the source of that cost is not yet - identified and needs an `--enable_wasm_profiling` build to attribute. -5. **Follow-up cleanup** listed at the end of the design doc, including removing the WebGPU special cases that - only exist while it is still an "internal" EP. -6. **Open the PR.** -7. **Keep the higher-level plan docs in sync with `edgchen1/webgpu_ep_extraction_plan`.** That branch is the - authoritative source for the overall extraction docs — the overview and the four workstream documents. It - deliberately does *not* carry `static_plugin_ep_registration_design.md` or this work log: it added them in - `a75b4c1294` and removed them again in `fdf2f53e0a`, so implementation-level docs live on this branch and the - new design docs are added when this branch lands. Until then, any change made here to a *higher-level* doc - must also be upstreamed to that branch. Two practical consequences: - - Do not merge this branch *into* the plan branch — that would restore the two files and undo `fdf2f53e0a`. - Merge in the other direction only, then upstream individual higher-level edits by hand. - - An edit that only adds a cross-reference to a doc that exists solely on this branch is *not* upstreamable - as-is; it would be a broken link there. Upstream the substantive part without the link, or defer it. - - Current delta against the plan branch, as of this entry: three hunks, all touching the static-plugin docs. - Two are pure cross-references and land with this branch. The third resolves a plan-level open question -- - static factories are registered by ORT core during `OrtEnv` creation, not before it and not through - environment construction options. That resolution is upstreamable on its own once the trailing link is - dropped; **decision: deferred until this branch lands**, at which point the link becomes valid and the whole - set can go across together. - -8. **Decide the latency tolerance — but reduce the cost first.** The native half of the latency criterion is now - measured and root-caused — see - [Native shared-library plugin measurements](#native-shared-library-plugin-measurements). Profiling showed the - overhead is redundant per-kernel tensor-wrapper construction in the EP API adapter, not intrinsic - boundary-crossing cost, so it looks reducible: caching the `Tensor` wrapper across `Compute()` calls, and - avoiding the allocator-name `std::string` and the `TensorShape` allocation on the per-node path, are the obvious - candidates. Negotiating a tolerance around a fixable defect would be the wrong order. Two gaps also remain in the - native numbers: session-creation cost was not compared (the plugin arm must also discover and load the DLL, so it - is likely worse in relative terms), and both arms ran with default WebGPU options — graph capture in particular - targets exactly the per-dispatch cost identified, and is worth a follow-up. - -9. **Restore `Api` profiling events on the plugin path.** The built-in WebGPU EP emits 15,300 `Api` category events - under `-p`; the plugin path emits none. This is an observability regression independent of latency, and it makes - ORT's own profiler unusable for comparing the two paths. - -## Environment notes - -Specific to the original machine, but the class of problem generalises. - -- The working copy `C:\source\ort-webgpu-static-plugin` is a **linked git worktree** of `C:\source\onnxruntime`. - `.git` is therefore a *file*, not a directory. Do not write scratch files into it. -- Linting is `lintrunner`, driven by the `ort-lint` skill. `lintrunner -a` can report "ok No lint issues" and - "Successfully applied all patches" in the same run, so always re-check `git status` afterwards. -- The repository's own pre-commit hook lives in `.githooks/pre-commit` (this is not the `pre-commit` framework) - and is enabled with `git config core.hooksPath .githooks`. It **silently does nothing if `lintrunner` is not - on `PATH`**, so activate the virtual environment before committing or the commit goes through unlinted. From 2ac50a16ef1cbcd7ec30097f841926b2491ec998 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 10 Sep 2026 18:11:50 -0700 Subject: [PATCH 070/107] Always pass the WebGPU plugin EP CMake flags explicitly onnxruntime_USE_EP_API_ADAPTERS and onnxruntime_WEBGPU_STATIC_PLUGIN were only emitted as =ON, from the --use_webgpu shared_lib/static_plugin branches. Since build.py never clears CMakeCache.txt, reusing a build directory kept a previous build's ON value: a subsequent --use_webgpu or --use_webgpu shared_lib build would silently produce a static plugin instead, with no error. Emit both unconditionally with an explicit ON/OFF, matching the treatment of onnxruntime_USE_WEBGPU just above. --cmake_extra_defines is appended afterwards, so it still overrides them. --- tools/ci_build/build.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tools/ci_build/build.py b/tools/ci_build/build.py index 930226b9663bf..f5cbcbbcf775c 100644 --- a/tools/ci_build/build.py +++ b/tools/ci_build/build.py @@ -559,6 +559,10 @@ def generate_build_tree( "-Donnxruntime_USE_ACL=" + ("ON" if args.use_acl else "OFF"), "-Donnxruntime_USE_JSEP=" + ("ON" if args.use_jsep else "OFF"), "-Donnxruntime_USE_WEBGPU=" + ("ON" if args.use_webgpu else "OFF"), + # The WebGPU EP library kind selects these. Emit them unconditionally, including the OFF case, so that + # reusing a build directory cannot leak a previous build's setting in via the CMake cache. + "-Donnxruntime_USE_EP_API_ADAPTERS=" + ("ON" if args.use_webgpu in ("shared_lib", "static_plugin") else "OFF"), + "-Donnxruntime_WEBGPU_STATIC_PLUGIN=" + ("ON" if args.use_webgpu == "static_plugin" else "OFF"), "-Donnxruntime_USE_EXTERNAL_DAWN=" + ("ON" if args.use_external_dawn else "OFF"), "-DDAWN_USE_AGILITY_SDK=" + ("ON" if args.use_dawn_agility_sdk else "OFF"), # Training related flags @@ -969,12 +973,8 @@ def generate_build_tree( ) elif args.use_webgpu == "shared_lib": # Shared library build (plugin EP) - cmake_args += ["-Donnxruntime_USE_EP_API_ADAPTERS=ON"] if args.build_wasm: raise BuildError("Only static library build of WebGPU EP is supported for WebAssembly build.") - elif args.use_webgpu == "static_plugin": - # Plugin EP that is linked into the ORT binary. - cmake_args += ["-Donnxruntime_USE_EP_API_ADAPTERS=ON", "-Donnxruntime_WEBGPU_STATIC_PLUGIN=ON"] if args.use_dawn_agility_sdk: if not is_windows(): From ad2413a5b6610f716ad996ee808e950640ac48db Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 10 Sep 2026 18:11:58 -0700 Subject: [PATCH 071/107] Add a build-only WASM static plugin EP CI leg Nothing in CI compiled the Emscripten static plugin EP registration path, so a break there would only surface locally. Add a --use_webgpu static_plugin build to the wasm_Release job, gated behind a new build_webgpu_static_plugin input. The output is not published. Its file names are identical to those of the published WebGPU build, and the ORT Web test harness picks the wasm binary by file name, so the two variants cannot ship side by side. Running the ORT Web suites against the plugin build waits on switching ORT Web over to it, at which point the existing web_Debug/web_Release lanes cover it and this leg becomes redundant. Update the design doc accordingly. --- .../linux-wasm-ci-build-and-test-workflow.yml | 22 +++++++++++++++++++ .github/workflows/web.yml | 1 + .../static_plugin_ep_registration_design.md | 20 +++++++++++------ 3 files changed, 36 insertions(+), 7 deletions(-) diff --git a/.github/workflows/linux-wasm-ci-build-and-test-workflow.yml b/.github/workflows/linux-wasm-ci-build-and-test-workflow.yml index b87c91e6029a6..84bd3d38e1683 100644 --- a/.github/workflows/linux-wasm-ci-build-and-test-workflow.yml +++ b/.github/workflows/linux-wasm-ci-build-and-test-workflow.yml @@ -34,6 +34,10 @@ on: required: false type: boolean default: false + build_webgpu_static_plugin: + required: false + type: boolean + default: false jobs: build-wasm: @@ -156,6 +160,24 @@ jobs: --skip_tests working-directory: ${{ github.workspace }} + # Build-only coverage for registering the WebGPU EP as a statically linked plugin EP under Emscripten. + # The output is deliberately not published: it has the same file names as the published WebGPU build, and + # the ORT Web test harness selects the wasm binary by file name, so both variants cannot ship together. + # This leg becomes redundant once the published WebGPU wasm build switches over to the plugin EP. + - name: Build (simd + threads + WebGPU experimental, static plugin EP) + if: ${{ inputs.build_webgpu_static_plugin == true }} + run: | + python ./tools/ci_build/build.py \ + ${{ env.common_build_args }} \ + --build_dir ${{ github.workspace }}/build/wasm_inferencing_webgpu_static_plugin \ + --use_webgpu static_plugin \ + --use_webnn \ + --target onnxruntime_webassembly \ + ${{ inputs.build_config == 'Release' && '--enable_wasm_api_exception_catching' || '' }} \ + ${{ env.reduced_size_build_args }} \ + --skip_tests + working-directory: ${{ github.workspace }} + - name: Create Artifacts if: ${{ inputs.skip_publish != true }} run: | diff --git a/.github/workflows/web.yml b/.github/workflows/web.yml index 0e2ba5e440756..25eac1871b13c 100644 --- a/.github/workflows/web.yml +++ b/.github/workflows/web.yml @@ -56,6 +56,7 @@ jobs: extra_build_args: "--target onnxruntime_webassembly --skip_tests --disable_rtti" build_jsep: true build_webgpu: true + build_webgpu_static_plugin: true wasm_Release_static_library: needs: precheck diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md index 317de001af34d..0fda4fb19a1fd 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -383,10 +383,14 @@ compiles and links cleanly, producing `ort-wasm-simd-threaded.asyncify.{mjs,wasm `static_plugin` is in fact the only plugin kind Emscripten supports — `cmake/onnxruntime_providers_webgpu.cmake` raises a `FATAL_ERROR` for `shared_lib` there, since there is no runtime library to load. -No CI leg covers this yet. The WASM WebGPU jobs pass a bare `--use_webgpu`, which defaults to `static_lib` -(`tools/ci_build/build_args.py` uses `const="static_lib"`), so they still build the internal EP. This change makes -the configuration available to ORT Web; switching ORT Web over to it is separate work, and has prerequisites that -this change does not address. See [ORT Web migration](#ort-web-migration). +The WASM WebGPU jobs that produce the published artifacts pass a bare `--use_webgpu`, which defaults to +`static_lib` (`tools/ci_build/build_args.py` uses `const="static_lib"`), so they still build the internal EP. +`.github/workflows/web.yml` additionally runs a **build-only** `--use_webgpu static_plugin` leg in the +`wasm_Release` job. It keeps the Emscripten registration path compiling, but deliberately publishes nothing: its +output file names are identical to those of the published WebGPU build, and the ORT Web test harness selects the +wasm binary by file name, so the two variants cannot ship side by side. Running the ORT Web suites against the +plugin build therefore waits on switching ORT Web over to it, which is separate work with prerequisites that this +change does not address. See [ORT Web migration](#ort-web-migration). A green build is *not* sufficient evidence that the registration is live. `CreateStaticPluginEpLibraries()` in `onnxruntime/core/session/plugin_ep/ep_static_plugins.cc` is guarded by @@ -550,12 +554,14 @@ A workable order: construction, and no additional build define or runtime capability probe is needed. What is *not* safe is landing the JS change without the build change, since the two must move together. -4. Flip the ORT Web WASM builds to `static_plugin` and add a CI leg. The two builds are +4. Flip the ORT Web WASM builds to `static_plugin` and extend CI to *test* the result. The two builds are `Build (simd + threads + WebGPU experimental)` and its JSPI sibling in `.github/workflows/linux-wasm-ci-build-and-test-workflow.yml`, both of which pass a bare `--use_webgpu` and so get `static_lib` from `build_args.py`'s `const="static_lib"`. Neither uses `--minimal_build`, so the - minimal-build `FATAL_ERROR` guard does not stand in the way of this migration. No pipeline builds WASM with - `static_plugin` today, so without a leg this path stays validated at the symbol level only. + minimal-build `FATAL_ERROR` guard does not stand in the way of this migration. A build-only `static_plugin` + leg already runs (see [Emscripten and ORT Web](#emscripten-and-ort-web)), so this path is covered at compile + time; flipping these two builds is what gets it *executed*, because the published artifacts then become the + plugin build and the existing `web_Debug` / `web_Release` suites cover it with no parallel test lane. *Prototype note:* the `js/web` test runner cannot currently exercise this configuration end to end, because `script/test-runner-cli.ts` spawns `script/build` with only `--bundle-mode` and does not forward `--webgpu-ep`. From 1aa7d1143781b7571205c63d9e7e0bd06e3ecfbf Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Fri, 11 Sep 2026 11:53:44 -0700 Subject: [PATCH 072/107] Correct factual errors in the static plugin EP design docs From documentation review feedback. D3 named ORT_PLUGIN_EP_OWNS_PROCESS_GLOBALS, which does not exist anywhere in the tree. The real guard is #if !defined(ORT_PLUGIN_EP_STATICALLY_LINKED) in onnxruntime/core/providers/webgpu/ep/api.cc. State the actual macro, its polarity and the ownership rationale, since the previous wording also inverted which build compiles the protobuf shutdown in. Background claimed ORT "has two EpLibrary implementations today" and that none accepts entry points directly. EpLibraryStaticPlugin now exists, so phrase this as the pre-change state. Resolve the OrtEnv::m_ condition variable question instead of leaving it open: m_ is a private static member, so all uses are confined to ort_env.cc and all are plain lock_guard acquisitions. Record it as an invariant. Also: soften the unprovable "byte-for-byte unchanged at runtime" goal to a statement about ABI and runtime behavior; give the non-WebGPU test plugin criterion an explicit disposition (deferred) rather than "where practical", noting that D2's prefix uniqueness contract is what stays untested; drop a trailing blockquote that restated D6; and restore the registration-timing question the workstream had replaced with a bare "Resolved: neither". --- ...boundary_and_web_integration_workstream.md | 11 ++++-- .../static_plugin_ep_registration_design.md | 37 ++++++++++++------- 2 files changed, 31 insertions(+), 17 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md index 5221b404b2383..2b67290af4df7 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md @@ -44,7 +44,8 @@ The design must address: - Dead-code elimination for statically linked providers. - Diagnostics for incompatible or duplicate registrations. -The facility must be generic and validated with at least one non-WebGPU test plugin where practical. +The facility must be generic. Validation with a non-WebGPU test plugin is deferred: WebGPU is currently the only +statically linked plugin EP, so this is tracked as a test gap rather than a completed criterion. The detailed design is in [Static Plugin EP Registration](static_plugin_ep_registration_design.md). @@ -182,9 +183,13 @@ the numbers can no longer be captured. the native shared-library plugin. - The direct `IExecutionProvider` WebGPU path is removed. +## Resolved decisions + +- Should static factories be registered before environment creation, or through environment construction options? + Neither. ORT core registers them during `OrtEnv` creation, after the environment is constructed and published. + See [Static Plugin EP Registration](static_plugin_ep_registration_design.md). + ## Open questions -- Resolved: neither. ORT core registers static factories during `OrtEnv` creation, after the environment is - constructed and published. See [Static Plugin EP Registration](static_plugin_ep_registration_design.md). - What is the stable representation of JavaScript-owned WebGPU objects at the C API boundary? - Which private-dependency findings require a public plugin EP API addition rather than a provider-owned replacement? diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md index 0fda4fb19a1fd..2c96503a4471c 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -1,6 +1,7 @@ # Static Plugin EP Registration -Status: Detailed design supporting +Status: Implemented in PR #32395. ORT Web's migration onto this path remains follow-up work; see +[ORT Web migration](#ort-web-migration). Detailed design supporting [Plugin Boundary and Web/Wasm Integration](plugin_boundary_and_web_integration_workstream.md). ## Purpose @@ -14,17 +15,18 @@ WebGPU is the first consumer. The immediate deliverable is to move the staticall ## Background -ORT has two `EpLibrary` implementations today: +Before this change, the two `EpLibrary` implementations relevant here were: - `EpLibraryPlugin` (`onnxruntime/core/session/plugin_ep/ep_library_plugin.cc`) loads a shared library, resolves the `CreateEpFactories` and `ReleaseEpFactory` symbols, and drives the factory lifecycle. - `EpLibraryInternal` (`onnxruntime/core/session/plugin_ep/ep_library_internal.cc`) wraps an in-tree `IExecutionProvider`, including `EpLibraryInternal::CreateWebGpuEp`. -There is no `EpLibrary` that accepts factory entry points directly, so a provider compiled into the host cannot reach -the plugin path. +Neither accepted factory entry points directly, so a provider compiled into the host could not reach the plugin +path. `EpLibraryStaticPlugin` (`onnxruntime/core/session/plugin_ep/ep_library_static_plugin.h`) is what this design +adds to close that gap. -This blocks ORT Web specifically. `cmake/onnxruntime_providers_webgpu.cmake` raises a `FATAL_ERROR` for the WebGPU +This blocked ORT Web specifically. `cmake/onnxruntime_providers_webgpu.cmake` raises a `FATAL_ERROR` for the WebGPU shared-module build under Emscripten, so static linking is the only way a Wasm build can ever reach the plugin boundary. @@ -35,7 +37,7 @@ boundary. - Present statically linked plugin code with the same runtime conditions as dynamically loaded plugin code. - Require no change to provider sources between the two linkage modes. - Avoid new host call sites for in-tree providers. -- Keep the shared-library path byte-for-byte unchanged at runtime. +- Keep the shared-library path's existing ABI and runtime behavior unchanged. ## Non-goals @@ -75,7 +77,12 @@ The shared build must continue to export the unprefixed `CreateEpFactories` and ### D3: No new EP library ABI hook for teardown All teardown remains in `ReleaseEpFactory`. Process-global shutdown that belongs to the host — currently -`google::protobuf::ShutdownProtobufLibrary()` — is compiled out under `ORT_PLUGIN_EP_OWNS_PROCESS_GLOBALS`. +`google::protobuf::ShutdownProtobufLibrary()` in `onnxruntime/core/providers/webgpu/ep/api.cc` — is guarded by +`#if !defined(ORT_PLUGIN_EP_STATICALLY_LINKED)`, so it is compiled **in** for the shared-library build and **out** +for the statically linked build. The polarity follows ownership: a shared plugin module owns the process-global +state it initialized and must tear it down, whereas a statically linked plugin shares protobuf with the host, which +owns its lifetime and shuts it down itself. `ORT_PLUGIN_EP_STATICALLY_LINKED` is defined by the static plugin +configuration only; see D7 for where it is defined and what else it controls. Rationale: a new optional entry point cannot be relied upon, because an older ORT would load a newer plugin and silently ignore it. Behavior that varies with the host version is worse than a compile-time decision. @@ -276,11 +283,18 @@ configuration entry, which remain in use for dynamically registered plugin libra dedicated single-test process or a test that fully releases and recreates the environment. - Whether any shipped ORT Web configuration uses an extended-minimal build, which excludes plugin EP infrastructure entirely. -- Whether any `std::condition_variable` is paired with `OrtEnv::m_`, which would require `condition_variable_any` - under a recursive mutex. +- No `std::condition_variable` is paired with `OrtEnv::m_`, so making it recursive does not require + `condition_variable_any`. This is an invariant to preserve rather than an open question: `m_` is a private static + member of `OrtEnv`, so every use is confined to `onnxruntime/core/session/ort_env.cc`, and all of them are plain + `std::lock_guard` acquisitions. A future `std::condition_variable` waiting on `m_` would not compile against a + recursive mutex, so the constraint is enforced by the type system rather than by review. - The existing rejection of `onnxruntime_BUILD_DAWN_SHARED_LIBRARY` together with the adapters. Initially left rejecting both plugin kinds. - Binary size and dead-code elimination for statically linked providers, per the workstream completion criteria. +- Validation with a non-WebGPU static plugin EP is deferred, not completed. `CreateStaticPluginEpLibraries()` + currently has exactly one entry, so the design is generic by construction but unexercised with more than one + provider. The specific untested contract is entry-point prefix uniqueness within a host binary (D2): with a + single provider a prefix collision cannot occur. This is a test gap, not a known defect. ## Validation status @@ -773,8 +787,3 @@ the `_OrtAppendExecutionProviderV2` call and the EP-name lookup are free and the `static_plugin` wasm's per-kernel C++ path. **Where exactly is not yet attributed** — an `--enable_wasm_profiling` build would give named frames, and the native root cause recorded under [Native shared-library plugin performance](#native-shared-library-plugin-performance) is the leading candidate. - -> Should static factories be registered before environment creation or through environment construction options? - -Neither. They are registered by ORT core during `OrtEnv` creation, after the environment is constructed and -published. See D6. From c642429e07706645475a4a946b976e8365015d67 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Fri, 11 Sep 2026 12:14:13 -0700 Subject: [PATCH 073/107] Split the static plugin EP design doc into design, validation and migration The 789-line document combined an architecture decision record, a test and CI report, an ORT Web migration design, and native and Wasm performance investigations. That buried the contract the document exists to define, and made it alternate between future, present and past tense. Keep the design document to the enduring contract and the reasoning behind it: purpose, background, goals, decisions D1-D7, an implementation map, follow-up cleanup, risks, and a short validation summary that links out. It is now 338 lines. Move the evidence to static_plugin_ep_registration_validation.md (test results, known coverage differences, CI jobs, symbol checks, native performance) and the ORT Web work to ort_web_static_plugin_migration.md (Emscripten build status, migration plan, reduced-size and size analysis, Wasm exception catching, web performance). Prose is carried across verbatim except for five internal anchors that became cross-document links, and the Implementation order list, which is replaced by a decision-to-code implementation map now that the work has landed. Add a decision summary table and a diagram showing where the shared and static paths converge. Correct the CI coverage section, which still said two jobs were added. --- .../ort_web_static_plugin_migration.md | 371 ++++++++++++ ...boundary_and_web_integration_workstream.md | 4 +- .../static_plugin_ep_registration_design.md | 555 ++---------------- ...tatic_plugin_ep_registration_validation.md | 138 +++++ 4 files changed, 564 insertions(+), 504 deletions(-) create mode 100644 docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/ort_web_static_plugin_migration.md create mode 100644 docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_validation.md diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/ort_web_static_plugin_migration.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/ort_web_static_plugin_migration.md new file mode 100644 index 0000000000000..69f533d104247 --- /dev/null +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/ort_web_static_plugin_migration.md @@ -0,0 +1,371 @@ +# ORT Web Static Plugin Migration + +Follow-up design for moving ORT Web onto the statically linked plugin EP path defined in +[Static Plugin EP Registration](static_plugin_ep_registration_design.md). Build and link evidence for the Emscripten +configuration, the remaining migration steps, and the size and performance analysis that constrains them. + +## Emscripten and ORT Web + +Making the statically linked WebGPU EP reachable through the plugin path is what motivates this design, so the +Emscripten configuration was built before landing the change: `--build_wasm --enable_wasm_simd +--enable_wasm_threads --use_webgpu static_plugin --use_webnn --target onnxruntime_webassembly`. It configures, +compiles and links cleanly, producing `ort-wasm-simd-threaded.asyncify.{mjs,wasm}`. + +`static_plugin` is in fact the only plugin kind Emscripten supports — `cmake/onnxruntime_providers_webgpu.cmake` +raises a `FATAL_ERROR` for `shared_lib` there, since there is no runtime library to load. + +The WASM WebGPU jobs that produce the published artifacts pass a bare `--use_webgpu`, which defaults to +`static_lib` (`tools/ci_build/build_args.py` uses `const="static_lib"`), so they still build the internal EP. +`.github/workflows/web.yml` additionally runs a **build-only** `--use_webgpu static_plugin` leg in the +`wasm_Release` job. It keeps the Emscripten registration path compiling, but deliberately publishes nothing: its +output file names are identical to those of the published WebGPU build, and the ORT Web test harness selects the +wasm binary by file name, so the two variants cannot ship side by side. Running the ORT Web suites against the +plugin build therefore waits on switching ORT Web over to it, which is separate work with prerequisites that this +change does not address. See [Migration plan](#migration-plan). + +A green build is *not* sufficient evidence that the registration is live. `CreateStaticPluginEpLibraries()` in +`onnxruntime/core/session/plugin_ep/ep_static_plugins.cc` is guarded by +`#if defined(USE_WEBGPU) && defined(ORT_WEBGPU_STATIC_PLUGIN)`; if that define does not reach ORT core the function +still compiles, simply returning an empty vector, and the build is just as green while registering nothing. The +check that actually discriminates is at the symbol level — ORT core must hold an *undefined* reference to the +prefixed entry points, resolved by the provider library: + +``` +llvm-nm libonnxruntime_session.a | grep WebGpu_ # U WebGpu_CreateEpFactories / U WebGpu_ReleaseEpFactory +llvm-nm libonnxruntime_providers_webgpu.a | grep WebGpu_ # T WebGpu_CreateEpFactories / T WebGpu_ReleaseEpFactory +``` + +Both were observed on the Emscripten build, which confirms the guard was live, the D3 entry-point prefixing worked, +and the link resolved core's reference against the provider. That check validates build, link and registration +wiring; runtime behaviour was established separately by the ORT Web prototype described under +[Migration plan](#migration-plan). + +A practical build note: the WASM build needs `node` on `PATH` — `node_helper.cmake` does a hard `find_program` and +fails configure without it. emsdk bundles a suitable one under `cmake/external/emsdk/node//bin`. + + +## Migration plan + +The intended end state is that the WebGPU EP is always a plugin EP and `--use_webgpu static_lib` goes away +entirely, ORT Web included. Building the EP into the WASM binary as a static plugin, which this change enables, is +only the first half of that. The second half is *selection*, and two gaps stand between the current state and a +working ORT Web on the plugin path. Neither is visible to the linker, so the symbol-level evidence above does not +speak to either. + +Both gaps have since been closed by a throwaway prototype, which ran the full `js/web` WebGPU operator suite in +Edge against a WASM `static_plugin` binary: 2152 tests, all passing. The design below is therefore validated end +to end rather than merely plausible; what remains is productionization, not discovery. The prototype's findings are +recorded inline in the steps that follow. + +**ORT Web selects the EP by name, and that name resolves only to the internal EP.** +`js/web/lib/wasm/session-options.ts` maps the `'webgpu'` backend to `epName = 'WebGPU'`, which reaches +`OrtAppendExecutionProvider` in `onnxruntime/wasm/api.cc` and from there +`OrtApis::SessionOptionsAppendExecutionProvider`. That API is a closed table of hardcoded EP names in +`onnxruntime/core/session/provider_registration.cc`, and its `EpID::WebGPU` case is compiled out under +`ORT_USE_EP_API_ADAPTERS`. On a `static_plugin` build the EP therefore registers at `OrtEnv` creation and is then +unselectable from JS: the call returns "WebGPU execution provider is not supported in this build". This is by +design rather than an oversight — the name-based API is the internal-EP mechanism, and plugin EPs are selected +through `RegisterExecutionProviderLibrary` plus `OrtEpDevice`-based selection — but it does mean ORT Web needs new +plumbing rather than a recompile. + +**No WebGPU `OrtEpDevice` exists under Emscripten.** Emscripten was neither `WIN32`, `LINUX` nor `APPLE`, so +`cmake/onnxruntime_common.cmake` fell through to `core/platform/device_discovery_default.cc`, which discovers only +a CPU device. `Factory::GetSupportedDevicesImpl` in `onnxruntime/core/providers/webgpu/ep/factory.cc` creates an +`OrtEpDevice` only for devices of type `OrtHardwareDeviceType_GPU`. The one fallback, the virtual GPU device in the +same function, is opt-in through the `allow_virtual_devices` environment configuration entry and is deliberately +registered without allocator info, because it exists to back a device-free compile-only session. So even after +selection plumbing is added, `GetEpDevices` would offer nothing for JS to select. + +A workable order: + +1. Make WebGPU visible as a device under Emscripten by adding an Emscripten device discovery implementation. + `cmake/onnxruntime_common.cmake` gains an `Emscripten` arm selecting a new + `core/platform/emscripten/device_discovery.cc`, which reports the CPU device via `GetCpuDeviceFromCPUIDInfo()` + plus a GPU `OrtHardwareDevice` when a synchronous `EM_ASM_INT` check finds `navigator.gpu`. No change to + `Factory::GetSupportedDevicesImpl` is needed: its existing loop filters on `OrtHardwareDeviceType_GPU`, so a + discovered GPU flows through and picks up both allocator infos unmodified. + + The prototype first tried the other option — having the factory synthesize the device itself through + `CreateHardwareDevice` — and it worked, but it is the wrong layer. `GetSupportedDevices` documents its `devices` + argument as "the `OrtHardwareDevice` instances that are available", so that array is the source of truth, and + `CreateHardwareDevice` is documented for devices with no hardware behind them ("e.g., virtual"). A browser GPU + is real hardware that ORT's discovery layer merely failed to enumerate, so the producer is the place to fix it. + Leaving it in the factory would also oblige every future Emscripten plugin EP to repeat the same workaround. + + A capability check is a legitimate form of discovery here. `core/platform/apple/device_discovery.cc` sets the + precedent: it enumerates nothing, it asserts that one GPU exists on Apple Silicon and hardcodes the vendor id. + `navigator.gpu` is a stronger signal than that — an actual runtime capability probe, and the same one ORT Web's + `initEp` uses to decide whether WebGPU is usable. `EM_ASM` from ORT core is likewise established practice + (`core/graph/model.cc`, `core/framework/external_data_loader.cc`), not just an EP-layer habit. + + The honest limitation is that no adapter information is available. `_OrtInit` creates the `OrtEnv` and therefore + triggers discovery, and it runs before `navigator.gpu.requestAdapter()` in ORT Web's `initEp`, so `vendor_id` + and `device_id` are left at 0. That is acceptable: Apple reports no device id either, and browsers mask + `GPUAdapterInfo.vendor` for fingerprinting resistance regardless. The device is deliberately *not* tagged with + `kOrtHardwareDevice_MetadataKey_IsVirtual`, which is what keeps it distinct from the virtual device — that path + is allocator-less by design and cannot back a real session. + + A later change on `main` added a second fallback to `GetSupportedDevicesImpl` alongside the virtual device: when + no GPU device is discovered, `allow_software_adapter` (opt-in via the `kAllowSoftwareAdapterEnvironmentVariable` + environment variable) advertises WebGPU against the *CPU* `OrtHardwareDevice`. It does not change the analysis + above. Both upstream fallbacks work around a device the discovery layer failed to enumerate; they are opt-in + precisely because misreporting a device is not safe to do by default. Under Emscripten the device is genuinely + there and cheaply detectable, so fixing discovery keeps the default path correct and needs no opt-in. The + primary matching loop that our discovered GPU device flows through is unchanged by that work. + + One accepted side effect: `ep_library_internal.cc` creates an internal `WebGpuEpFactory` in a + `USE_WEBGPU && !ORT_USE_EP_API_ADAPTERS` build, so today's `static_lib` ORT Web build starts producing an + `OrtEpDevice` where it previously produced none. That build selects by name and is slated for removal, so the + change is benign. + +2. Expose plugin EP selection in the WASM C API. `g_env` is already available in `api.cc`, so `GetEpDevices` and + `SessionOptionsAppendExecutionProvider_V2` are both reachable. A narrow helper that appends by EP name and + resolves the device internally avoids marshalling arrays of device pointers across the JS boundary. The new + entry point needs to be exported from `cmake/onnxruntime_webassembly.cmake`. + + *Prototype result:* the name-resolving helper is the right shape. Note that exporting it takes three edits, not + one: `EMSCRIPTEN_KEEPALIVE` in `api.h`, the `JSPI_EXPORTS` list in `cmake/onnxruntime_webassembly.cmake` for + JSPI builds, and the `wrapAsyncAPIs` list in `onnxruntime/wasm/pre-async.js` for Asyncify builds. Omitting the + last one is the easy mistake, and Asyncify is the default variant. Reporting matters too: on no match the helper + should raise a status naming the EPs that *are* registered, which is what makes a misconfiguration diagnosable + from JS through `checkLastError`. + +3. Switch `js/web` to the new call. The `BUILD_DEFS.DISABLE_WEBGPU` branch that falls back to the JS EP is + unaffected. Confirm that the EP options ORT Web passes today — `deviceId`, `webgpuInstance`, `webgpuDevice`, + `preferredLayout` and the buffer cache modes — still reach the provider, since `Factory::CreateEpImpl` obtains + them from the session config options. + + *Prototype result:* the options do reach the provider unchanged, and the reason is worth recording. Both paths + derive their config prefix from `OrtSessionOptions::GetProviderOptionPrefix` keyed on the EP name, so V2 lands + the options under exactly the same `ep.webgpuexecutionprovider.*` keys the name-based path used. One JS change + is required though: `session-options.ts` passes the short name `'WebGPU'`, whereas `OrtEpDevice` selection + matches the canonical `'WebGpuExecutionProvider'` returned by `EpDevice_EpName`. + + The productionization question this raises is *conditionality*: is it safe to switch to V2 unconditionally + whenever WebGPU is enabled? It is, provided the WASM build flips at the same time. `BUILD_DEFS.DISABLE_WEBGPU` + is defined in `js/web/script/build.ts` as exactly `!USE_WEBGPU_EP`, so the bundle is already specialized per + WASM flavour by `--webgpu-ep` — the JSEP bundle takes the other branch entirely and is unaffected. Flipping the + WebGPU WASM builds to `static_plugin` (step 4) therefore makes `!DISABLE_WEBGPU` imply `static_plugin` by + construction, and no additional build define or runtime capability probe is needed. What is *not* safe is + landing the JS change without the build change, since the two must move together. + +4. Flip the ORT Web WASM builds to `static_plugin` and extend CI to *test* the result. The two builds are + `Build (simd + threads + WebGPU experimental)` and its JSPI sibling in + `.github/workflows/linux-wasm-ci-build-and-test-workflow.yml`, both of which pass a bare `--use_webgpu` and so + get `static_lib` from `build_args.py`'s `const="static_lib"`. Neither uses `--minimal_build`, so the + minimal-build `FATAL_ERROR` guard does not stand in the way of this migration. A build-only `static_plugin` + leg already runs (see [Emscripten and ORT Web](#emscripten-and-ort-web)), so this path is covered at compile + time; flipping these two builds is what gets it *executed*, because the published artifacts then become the + plugin build and the existing `web_Debug` / `web_Release` suites cover it with no parallel test lane. + + *Prototype note:* the `js/web` test runner cannot currently exercise this configuration end to end, because + `script/test-runner-cli.ts` spawns `script/build` with only `--bundle-mode` and does not forward `--webgpu-ep`. + Setting `npm_config_webgpu_ep` in the environment works as a stopgap, since `script/build.ts` reads it, but a CI + leg should forward the flag properly. Note also that without `--webgpu-ep` the `webgpu` backend silently routes + through JSEP instead, so a leg that omits it would pass while testing nothing relevant. + +## Interaction with the ORT Web reduced-size build options + +The ORT Web WASM builds are size-sensitive and pass a set of size-reduction flags that the local prototype builds +did not. `linux-wasm-ci-build-and-test-workflow.yml` applies `--disable_ml_ops --disable_generation_ops +--disable_types string float4 float8 optional sparsetensor --include_ops_by_config +onnxruntime/wasm/reduced_types.config --enable_reduced_operator_type_support`, and `web.yml` adds `--disable_rtti` +for the release job. Since the migration changes how the WebGPU EP is compiled and linked, each of these was +checked against the plugin path. + +**None of them need adapting.** The findings: + +- *Compile definitions reach the WebGPU target in every mode.* `onnxruntime_providers_webgpu` is created through + `onnxruntime_add_static_library` / `onnxruntime_add_shared_library_module`, both of which call + `onnxruntime_configure_target` → `onnxruntime_set_compile_flags`. That function is where every `DISABLE_*` and + `REDUCED_OPS_BUILD` definition is applied, so the `static_lib`, `static_plugin` and shared-library builds all + receive an identical set. There is no propagation gap introduced by the plugin boundary. + +- *Operator and type reduction never applied to WebGPU, before or after.* + `op_registration_utils.get_kernel_registration_files()` hardcodes the CPU registration files (plus CUDA when + requested); WebGPU is not in the list, so `--include_ops_by_config` does not rewrite WebGPU kernel + registrations. `onnxruntime/wasm/reduced_types.config` is `!no_ops_specified_means_all_ops_are_required` — it + performs global *type* reduction only, with no op exclusion — and that mechanism works through the + `op_kernel_type_control` macros, which neither `core/providers/webgpu` nor `contrib_ops/webgpu` uses. Its saving + is entirely CPU-EP-side and is unchanged by the plugin switch. This is a pre-existing gap, not a regression. + +- *The type and ML/generation op flags are inert for WebGPU.* No source under `core/providers/webgpu` or + `contrib_ops/webgpu` references `DISABLE_ML_OPS`, `DISABLE_GENERATION_OPS`, `DISABLE_SPARSE_TENSORS`, + `DISABLE_OPTIONAL_TYPE`, `DISABLE_FLOAT8_TYPES`, `DISABLE_FLOAT4_TYPES` or `DISABLE_STRING_TYPE`. + +- *`--disable_rtti` is safe on the plugin path.* Neither `include/onnxruntime/ep` (the EP API adapters) nor + `onnxruntime/core/session/plugin_ep` uses `dynamic_cast` or `typeid`. + +One consequence is worth recording for the *shared library* plugin EP, which is out of scope here but shares this +code. `onnxruntime_c_api.h` contains no `DISABLE_*` guards, so the `OrtApi` struct layout is invariant under these +flags and the ABI is stable. `onnxruntime_cxx_api.h` does guard some `Ort::Value` members (the sparse tensor +methods) — an EP DLL built with a different `--disable_types` than its host would therefore see a different C++ +header surface, but since those wrappers are inline over stable function pointers this is a source-compatibility +concern rather than an ABI one. For `static_plugin` everything is compiled in one tree with one set of defines, so +the question does not arise. + +**What is *not* covered by any existing flag** is the plugin machinery itself. Relative to `static_lib`, the +`static_plugin` build adds `core/providers/webgpu/ep/*` (excluded from the source list in the `static_lib` branch +of `onnxruntime_providers_webgpu.cmake`), inlines the header-only EP API adapters into every WebGPU translation +unit, and reaches the `OrtGraph` / `OrtNode` C API graph views through `OrtEp::GetCapability`. Note that +`core/session/plugin_ep` is *not* part of that delta: `onnxruntime_session.cmake` only excludes it for +`onnxruntime_MINIMAL_BUILD`, so today's WASM builds already compile it in. + +That also means the only existing lever that trims plugin-EP machinery is `--minimal_build`, which +`static_plugin` currently rejects outright. If measurement shows the size delta matters, lifting that restriction +(already listed as a follow-up) is the mechanism to reach parity — not a new size-reduction flag. + +### Measured size comparison + +The ORT Web npm package ships **two** WebGPU WASM artifacts, not one, and both were measured. The release path +(`npm-packaging-pipeline.yml` → `templates/web-ci.yml` → `templates/linux-wasm-ci.yml`) and the GitHub Actions path +(`web.yml` → `linux-wasm-ci-build-and-test-workflow.yml`) pass an identical flag set for the WebGPU legs, which +differ from each other only in their exception/stack-switching mode: + +| Leg | Artifact | Distinguishing flag | +| --- | --- | --- | +| `wasm_inferencing_webgpu` | `ort-wasm-simd-threaded.asyncify.{wasm,mjs}` | `--enable_wasm_api_exception_catching` | +| `wasm_inferencing_webgpu_jspi` | `ort-wasm-simd-threaded.jspi.{wasm,mjs}` | `--enable_wasm_jspi` | + +Four clean from-scratch builds were run (both EP modes × both legs) with the exact release flag set: `Release`, +SIMD + threads, WebNN on, `--disable_rtti`, `--target onnxruntime_webassembly`, and the full reduced-size set +above. Compressed sizes are included because that is what is actually delivered over the wire; they were produced +with gzip level 9 and brotli quality 11. + +`ort-wasm-simd-threaded.asyncify.wasm`: + +| Metric | `static_lib` | `static_plugin` | Delta | +| --- | ---: | ---: | ---: | +| raw | 26,549,423 B | 26,487,397 B | **−62,026 B (−0.234%)** | +| gzip | 6,537,701 B | 6,566,088 B | **+28,387 B (+0.434%)** | +| brotli | 3,953,625 B | 3,971,550 B | **+17,925 B (+0.453%)** | + +`ort-wasm-simd-threaded.jspi.wasm`: + +| Metric | `static_lib` | `static_plugin` | Delta | +| --- | ---: | ---: | ---: | +| raw | 16,627,577 B | 16,687,113 B | **+59,536 B (+0.358%)** | +| gzip | 4,099,914 B | 4,126,147 B | **+26,233 B (+0.640%)** | +| brotli | 2,651,643 B | 2,674,520 B | **+22,877 B (+0.863%)** | + +The accompanying `.mjs` files are byte-identical between the two EP modes in both legs (53,300 B asyncify, 51,204 B +JSPI); their compressed sizes differ by at most 23 B. + +**Every delta is under 1%, which is the substantive conclusion.** Two details are worth recording so the numbers +are not over-read: + +- *The raw sign is not stable across the two artifacts.* `static_plugin` is marginally smaller raw on asyncify but + marginally larger raw on JSPI. An earlier revision of this document reported only the asyncify raw number and + generalised it to "no size regression"; that generalisation was too broad. The asyncify measurement itself + reproduces well — it was −61,968 B when first taken and −62,026 B in this from-scratch rerun at a later commit. + +- *Compressed size grows slightly in both legs*, by +0.4% to +0.9%, so the direction is consistent once the + artifacts are compressed. Why compressed size can rise while raw size falls was not investigated; no + symbol-level or per-section attribution was done, and none of the earlier speculation about *which* code the + plugin build drops or adds has been measured. Given the magnitude, this was not pursued further. + +Both configurations compiled cleanly with every reduced-size flag in both legs, which empirically confirms the +compatibility analysis above. For reference, the same `static_plugin` asyncify configuration without any +reduced-size flags is 40.9 MB, so those flags are worth ~15 MB and remain essential regardless of EP mode. + +### `--enable_wasm_api_exception_catching` and the EP API adapters + +The one option that *did* interact with the plugin path is `--enable_wasm_api_exception_catching`. It is not a +size-reduction flag in the same sense as the others, but it is part of the same CI flag set and it exposed a real +bug. + +`onnxruntime_webassembly.cmake` compiles only `wasm/api.cc` and `core/session/onnxruntime_c_api.cc` with +`-sDISABLE_EXCEPTION_CATCHING=0`; every other translation unit keeps Emscripten's default, where `catch` clauses +are compiled so that they never match. Exceptions still propagate — they are simply only catchable at the C API +boundary. + +The EP API adapters in `include/onnxruntime/ep/adapter/` are header-only and are therefore inlined into WebGPU +translation units that have catching disabled. `OpKernelInfo::GetAttr()` / `GetAttrs()` were implemented as a +`try` around the throwing `Ort::` C++ wrappers, converting `Ort::Exception` into a `Status`. That makes a *missing +optional attribute* — ordinary control flow for `GetAttrOrDefault()` — depend on catching an exception. With +catching disabled the `catch` never ran, so the exception escaped `GetAttrOrDefault()` entirely and surfaced from +session creation as e.g. `ERROR_CODE: 6, ERROR_MESSAGE: No attribute with name:'extrapolation_value'is defined.` + +This reproduced as 949 `suite0` failures on the reduced-size `static_plugin` build while the same build passed on +the CPU backend, the reduced-size `static_lib` build passed, and the full-op `static_plugin` build passed — the +failure required plugin + WebGPU + API-only exception catching together. + +The fix is to not use exceptions for control flow in the adapters: `GetAttr()` / `GetAttrs()` now call the +non-throwing `OrtApi::KernelInfoGetAttribute*` functions directly and convert the returned `OrtStatus*` into a +`Status`. **General rule: header-only EP API adapter code must not rely on catching exceptions**, because a plugin +EP may be compiled with exception catching disabled. The remaining `catch` blocks in `include/onnxruntime/ep` +(`common.h`, `get_capability_utils.h`, `adapter/kernel_registry.h`) sit on genuinely exceptional callback +boundaries rather than on control-flow paths; when catching is disabled those exceptions still reach the outer C +API boundary and are reported, only with a less specific message. + +With that fix the reduced-size `static_plugin` build passes `suite0` on the WebGPU backend in full: 2152 tests, +all passing, matching the `static_lib` control run on the identical flag set. + +### Measured performance comparison + +The same two binaries were compared on inference performance in Edge, using the test runner's `--perf` mode +(`test-runner-cli model -b=webgpu -e=edge -P=`) against two synthetic models chosen to isolate the two +places the plugin path could plausibly cost something: + +- **dispatch-bound** — 300 chained tiny elementwise ops (`Mul`/`Add`/`Sub`) on a `[1, 1024]` tensor. Almost no GPU + work per node, so the run time is dominated by per-node host-side work, which is exactly what now goes through + the C API adapters. +- **compute-bound** — 16 chained `[512, 512]` `MatMul`s. Host-side per-node cost is negligible against the GPU + work. + +Three runs per model per build. **The two builds must be sampled interleaved** — one round is +`plugin/dispatch, plugin/compute, lib/dispatch, lib/compute`, repeated — rather than all samples of one build +followed by all samples of the other. This machine drifts: a build that has just finished, a warm browser, or a +warm GPU shifts timings by more than the effect being measured, and sequential sampling aliases that drift onto +the build under test. The table reports the median of the three per-run P50 values. + +| Metric | `static_lib` | `static_plugin` | Delta | +| --- | ---: | ---: | ---: | +| dispatch-bound, per-run P50 | 11.20 ms | 12.00 ms | +0.8 ms (+7%) | +| compute-bound, per-run P50 | 7.70 ms | 7.80 ms | +0.1 ms (within noise) | + +Reading of these numbers: + +- **Execution is not measurably slower once GPU work dominates.** The compute-bound samples overlap + (`static_plugin` 7.80 / 7.80 / 7.40 against `static_lib` 7.90 / 7.00 / 7.70 — the plugin build's best sample + beats two of the three control samples), so the difference is not resolvable at this sample count. +- **A dispatch-bound graph pays roughly 7%.** That is the cost of routing per-node kernel work through the C API + adapters instead of a direct in-process virtual call. Unlike the compute-bound case this one does separate + cleanly — the plugin build's slowest control sample is still faster than its own fastest sample. It only shows + up when nodes do essentially no GPU work, which is the worst case by construction rather than a representative + one. Amortized over ~300 nodes it is on the order of 3 µs per node — but see the caveat below before relying on + that figure. +- **Session creation costs about 3%** (measured pre-merge: 1136 → 1172 ms and 1250 → 1290 ms for the two models). + This covers plugin EP registration, `GetCapability` and kernel creation through the C API. The absolute figure + is dominated by fetching and parsing the model, so the true relative cost of the plugin machinery within session + creation is higher than 3% — but it is tens of milliseconds once, not per inference. + +Two cautions on methodology: + +- The *total* wall time the test runner prints for `suite0` is not a usable metric. It varied between 3 min 15 s + and 4 min 04 s for the **same** `static_lib` binary across two runs. Only the second figure it prints — net test + time, excluding the per-model `before all` session-creation hooks — is stable enough to compare, and by that + measure the two builds are equal. An early single-sample comparison of the total figure suggested a 29% + regression that does not exist. +- Timings taken with `--webgpu.profiling.mode=default` are inflated by the timestamp queries and must never be + compared against timings taken without it. + +**The per-node magnitude is not reliably established, and a controlled re-measurement is outstanding.** Three +separate measurements of the same 300-node dispatch-bound model disagree by roughly 3.7x: +0.80 ms (2.7 µs/node) +above, +1.50 ms (5.0 µs/node) in a later four-model sweep that fitted 4.76 µs/node across 150/300/600 nodes, and ++3.00 ms (10.0 µs/node) in a four-side build comparison. The `static_lib` baseline itself moved from 7.90 ms to +11.90 ms for the same model on the same machine within a day, so build-to-build and session drift dominates the +spread, and Edge's 0.1 ms `performance.now()` floor without cross-origin isolation limits resolution further. The +three runs also did not use the same statistic (per-run P50 versus median-of-round-minimum). A re-measurement +should record the commit of each arm, the full build flag set, the model definitions, the statistic, the +interleaved sampling order and the GPU submit counts, so the number can be reproduced. + +What *is* robust is the sign and the mechanism. The plugin side was slower in 26 of 26 paired rounds in the sweep +and 6 of 6 in a three-way run, and GPU submit counts were byte-identical on every side, model and round, so the +cost is CPU-side per-kernel work rather than extra queue work. The same three-way run also excludes the +JavaScript side: a `mismatch` arm (plugin bundle against a `static_lib` wasm) came out level with `static_lib`, so +the `_OrtAppendExecutionProviderV2` call and the EP-name lookup are free and the cost is inside the +`static_plugin` wasm's per-kernel C++ path. **Where exactly is not yet attributed** — an `--enable_wasm_profiling` +build would give named frames, and the native root cause recorded under [Native shared-library plugin +performance](static_plugin_ep_registration_validation.md#native-shared-library-plugin-performance) is the leading +candidate. diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md index 2b67290af4df7..8efe3e0b0d146 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md @@ -47,7 +47,9 @@ The design must address: The facility must be generic. Validation with a non-WebGPU test plugin is deferred: WebGPU is currently the only statically linked plugin EP, so this is tracked as a test gap rather than a completed criterion. -The detailed design is in [Static Plugin EP Registration](static_plugin_ep_registration_design.md). +The detailed design is in [Static Plugin EP Registration](static_plugin_ep_registration_design.md), with supporting +evidence in [Static Plugin EP Registration: Validation](static_plugin_ep_registration_validation.md) and the ORT Web +follow-up in [ORT Web Static Plugin Migration](ort_web_static_plugin_migration.md). ## Process-global ownership and teardown diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md index 2c96503a4471c..bfb6528c57059 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -1,7 +1,7 @@ # Static Plugin EP Registration Status: Implemented in PR #32395. ORT Web's migration onto this path remains follow-up work; see -[ORT Web migration](#ort-web-migration). Detailed design supporting +[ORT Web Static Plugin Migration](ort_web_static_plugin_migration.md). Detailed design supporting [Plugin Boundary and Web/Wasm Integration](plugin_boundary_and_web_integration_workstream.md). ## Purpose @@ -13,6 +13,29 @@ boundary. WebGPU is the first consumer. The immediate deliverable is to move the statically linked WebGPU build off `EpLibraryInternal` and onto the plugin EP path. +## Decision summary + +| Concern | Decision | +| --- | --- | +| D1 Discovery | ORT core owns a hand-written, build-guarded static registry | +| D2 Symbol names | Shared builds use standard names; static builds use provider-prefixed names | +| D3 Process globals | Ownership is selected at build time; no new optional teardown ABI | +| D4 Build configuration | `onnxruntime_WEBGPU_STATIC_PLUGIN` selects linked-in plugin mode | +| D5 Test infrastructure | The plugin library path becomes optional | +| D6 Registration timing | Register after publishing `OrtEnv`; environment creation stays serialized | +| D7 C++ API initialization | Linked-in plugins use the host binary's normal initialization mode | + +Both linkage modes converge on one factory lifecycle; only discovery differs: + +```mermaid +flowchart LR + Shared[Shared plugin module] --> Load[Load library, resolve symbols] + Static[Static plugin registry] --> Direct[Use linked entry points] + Load --> Factories[Shared factory lifecycle] + Direct --> Factories + Factories --> Devices[Register factories and devices in Environment] +``` + ## Background Before this change, the two `EpLibrary` implementations relevant here were: @@ -238,20 +261,24 @@ so the EP's own API access is unchanged. The macro is generic rather than WebGPU-specific because `ep/api.h` is shared with the CUDA plugin EP. -## Implementation order - -1. `EpLibraryStaticPlugin`, factoring the shared factory lifecycle out of `EpLibraryPlugin`. -2. The static registry and `Environment::CreateAndRegisterStaticPluginEps`, with the `ort_env.cc` - publish-then-register change. -3. `ep/api.cc` entry point prefixing and host-owned process-global guards. -4. Build option and `build.py` changes. -5. Test infrastructure changes. -6. `ORT_PLUGIN_EP_STATICALLY_LINKED` in `include/onnxruntime/ep/api.h`, per D7. -7. Applying the WebGPU plugin EP test definitions to `onnxruntime_provider_test` as well as `onnxruntime_test_all`, - via a shared `onnxruntime_set_webgpu_plugin_ep_test_definitions` cmake function. `onnxruntime_provider_test` holds - the operator tests, so without this the plugin path has no operator test coverage. It also needs - `ORT_UNIT_TEST_HAS_WEBGPU_STATIC_PLUGIN_EP` in its own right, because that macro is what enables the - `allow_virtual_devices` environment configuration entry at environment creation in `test_main.cc`. +## Implementation map + +| Decision | Owning code | +| --- | --- | +| D1 static registry | `core/session/plugin_ep/ep_static_plugins.cc`, `ep_library_static_plugin.h` | +| D1 shared factory lifecycle | `core/session/plugin_ep/ep_library_plugin.cc`, factored out for reuse | +| D2 entry point prefixing | `core/providers/webgpu/ep/api.cc`, `include/onnxruntime/ep/api.h` | +| D3 process-global ownership | `core/providers/webgpu/ep/api.cc` `ReleaseEpFactory` | +| D4 build option | `cmake/CMakeLists.txt`, `cmake/onnxruntime_providers_webgpu.cmake`, `tools/ci_build/build.py` | +| D5 optional library path | `test/unittest_util/test_dynamic_plugin_ep.cc` | +| D6 registration timing | `core/session/ort_env.cc`, `Environment::CreateAndRegisterStaticPluginEps` | +| D7 C++ API initialization | `include/onnxruntime/ep/api.h` | + +Test definitions are applied to `onnxruntime_provider_test` as well as `onnxruntime_test_all`, via a shared +`onnxruntime_set_webgpu_plugin_ep_test_definitions` cmake function. `onnxruntime_provider_test` holds the operator +tests, so without this the plugin path has no operator test coverage. It also needs +`ORT_UNIT_TEST_HAS_WEBGPU_STATIC_PLUGIN_EP` in its own right, because that macro is what enables the +`allow_virtual_devices` environment configuration entry at environment creation in `test_main.cc`. ## Follow-up cleanup @@ -296,494 +323,16 @@ configuration entry, which remain in use for dynamically registered plugin libra provider. The specific untested contract is entry-point prefix uniqueness within a host binary (D2): with a single provider a prefix collision cannot occur. This is a test gap, not a known defect. -## Validation status - -Verified on Windows with `--use_webgpu static_plugin`, Debug: - -- `onnxruntime_test_all` and `onnxruntime_provider_test` build and link. -- The statically linked WebGPU plugin EP is registered by ORT core during `Ort::Env` creation, and its `OrtEpDevice` - is enumerated and selectable, confirming the D6 publish-then-register ordering resolves the self-deadlock. -- `InferenceSessionTests.WebGpuVirtualDeviceCompileOnlyEndToEnd` and - `InferenceSessionTests.WebGpuVirtualDeviceRejectedWithoutCompileOnly` pass, which covers session creation and EP - instantiation end to end through the public V2 API on the static plugin path. - -Operator execution was initially not verifiable locally: the original development machine was a virtual machine with -no real GPU, so only the virtual WebGPU device was surfaced and non-compile-only sessions were rejected up front. This -was confirmed to be a property of the host rather than of static linking by building the shared library plugin -configuration (`--use_webgpu shared_lib`) on the same host and running the same tests: it passed and failed exactly -the same set (`WebGpuVirtualDeviceCompileOnlyEndToEnd` and `WebGpuVirtualDeviceRejectedWithoutCompileOnly` pass; -`WebGpuCompileOnlySkipsFinalization` and `TestStrictShapeInference` fail with the identical virtual-device rejection -message). - -This gap has since been closed on a machine with a real GPU (NVIDIA RTX 5060 Ti). The `static_plugin` and internal-EP -configurations were built from the same tree, RelWithDebInfo, differing only in the EP path, and their test results -compared: - -- `onnxruntime_provider_test`: 0 failures in both. **Across the 5894 tests common to both builds there were zero - status differences.** -- `onnxruntime_test_all`: 1965 passed in both, no differences. All three `InferenceSessionTests.WebGpu*` pass on the - static plugin build, including `WebGpuCompileOnlySkipsFinalization`, confirming its earlier failure was indeed a - property of the GPU-less host. - -### Known test coverage difference on the plugin path - -`cmake/onnxruntime_unittests.cmake` gates the `test/providers/webgpu/*` sources on -`onnxruntime_USE_WEBGPU AND NOT onnxruntime_USE_EP_API_ADAPTERS`, so those tests are **absent** from any adapter -build, including `static_plugin`. They are white-box tests that include internal WebGPU EP headers and therefore -cannot compile against the adapter boundary by construction. - -Of the 44 tests this excludes, 16 are `DISABLED_` and never run, and most of the rest assert on EP internals -(`WebGpuContextTest`, `ActivationCacheKeyTest`). However, 12 of them — `HardSwish_WebGPU` and `MatMul2BitsWebGpu` — -are genuine operator tests, and they currently run **only** on the internal EP path. This is a known coverage gap of -the plugin path. Closing it would mean rewriting them as EP-agnostic operator tests so both paths execute them; that -is deliberately left out of this change to keep it focused. - -Separately, the `InferenceSessionTests.WebGpu*` virtual-device tests are deliberately *paired* rather than shared -between the two builds, since each drives a different factory. `WebGpuVirtualDeviceCompileOnlyEndToEnd` and -`WebGpuVirtualDeviceRejectedWithoutCompileOnly` are the plugin-build counterparts of `WebGpuEpFactoryVirtualDevice` -and `WebGpuEpFactoryRejectsVirtualDeviceWithoutCompileOnly`. Only `WebGpuCompileOnlyUsesNoOpAllocator` has no -counterpart, because it asserts an internal allocator *type* through `dynamic_cast`, which is inherently -unobservable across the ABI boundary. - -The shared library plugin configuration was also re-verified after the D7 change, since `include/onnxruntime/ep/api.h` -is shared with the CUDA plugin EP. It builds, and `onnxruntime_providers_webgpu.dll` still exports exactly the -unprefixed `CreateEpFactories` and `ReleaseEpFactory` required by `EpLibraryPlugin::Load`, per D2. - -### CI coverage - -No pre-existing CI leg built this configuration, since both new CMake options default to `OFF`. Two jobs were added: - -- `webgpu_static_plugin_build_x64_RelWithDebInfo` in `.github/workflows/windows_webgpu.yml`, which builds and runs the - tests on a GPU-equipped pool. This closes the operator execution gap above. Note that it must not set - `onnxruntime_BUILD_DAWN_SHARED_LIBRARY=ON`, which is incompatible with `onnxruntime_USE_EP_API_ADAPTERS`. -- `build-linux-webgpu-static-plugin-x64-release` in `.github/workflows/linux_webgpu.yml`, build only, mirroring its - sibling job. This is the only GCC coverage of the EP adapters, and the only coverage of the `onnxruntime_python.cmake` - change, hence `--build_wheel` is kept. - -The first run of the Linux job surfaced `-Werror=maybe-uninitialized` errors. The EP adapter `OpKernelInfo` defines its -attribute accessors inline in the header, so GCC can see their failure and exception paths, whereas the in-tree -`OpKernelInfo` hides them behind an out-of-line definition. Any value that is only read when the attribute lookup -succeeded must be value-initialized. - -More generally, this job is the first time the EP adapters have ever been compiled with GCC, and it has surfaced -several successive batches of `-Werror` diagnostics — including false positives inside `absl::InlinedVector` and, in -optimized builds, `-Werror=array-bounds` on `std::string` concatenation chains that the adapter headers cause to be -inlined more aggressively. Because ninja stops at the first failure and reports only the targets already in flight, -**each CI run reveals just one batch**. When touching this configuration, enumerate the full set locally in a single -pass — an AlmaLinux 8 container with `gcc-toolset-14` reproduces the CI compiler exactly — rather than discovering -them one CI round trip at a time. Fixes that are correct on their own merits are applied at the source; irreducible -false positives use a suppression scoped to the `onnxruntime_providers_webgpu` target, GCC only, and conditional on -`onnxruntime_USE_EP_API_ADAPTERS`, so non-adapter builds stay strict. This mirrors what the CUDA plugin EP does in -`cmake/onnxruntime_providers_cuda_plugin.cmake`. - -That full local enumeration has since been done, and it is worth recording what it found: of the ~340 warnings GCC 14 -emits across the whole `static_plugin` build, **exactly one is attributable to ORT's own source** — the -`-Warray-bounds` false positive in `contrib_ops/webgpu/quantization/matmul_nbits_mlp.cc`, fixed by building the -expression with `MakeString` instead of a `std::string` concat chain. Everything else lives in Dawn, ONNX, -flatbuffers, or the telemetry dependency, in targets that do not use `-Werror`. The practical lesson is that the -adapters do not make ORT's code broadly warning-prone; they perturb inlining at a small number of specific sites, -so a targeted source fix is almost always available and preferable to a blanket target-wide suppression. - -When validating such a fix locally, always rebuild the *unfixed* source through the same harness first and confirm it -still errors. A harness that silently fails to exercise the warning is indistinguishable from a successful fix. - -### Emscripten and ORT Web - -Making the statically linked WebGPU EP reachable through the plugin path is what motivates this design, so the -Emscripten configuration was built before landing the change: `--build_wasm --enable_wasm_simd ---enable_wasm_threads --use_webgpu static_plugin --use_webnn --target onnxruntime_webassembly`. It configures, -compiles and links cleanly, producing `ort-wasm-simd-threaded.asyncify.{mjs,wasm}`. - -`static_plugin` is in fact the only plugin kind Emscripten supports — `cmake/onnxruntime_providers_webgpu.cmake` -raises a `FATAL_ERROR` for `shared_lib` there, since there is no runtime library to load. - -The WASM WebGPU jobs that produce the published artifacts pass a bare `--use_webgpu`, which defaults to -`static_lib` (`tools/ci_build/build_args.py` uses `const="static_lib"`), so they still build the internal EP. -`.github/workflows/web.yml` additionally runs a **build-only** `--use_webgpu static_plugin` leg in the -`wasm_Release` job. It keeps the Emscripten registration path compiling, but deliberately publishes nothing: its -output file names are identical to those of the published WebGPU build, and the ORT Web test harness selects the -wasm binary by file name, so the two variants cannot ship side by side. Running the ORT Web suites against the -plugin build therefore waits on switching ORT Web over to it, which is separate work with prerequisites that this -change does not address. See [ORT Web migration](#ort-web-migration). - -A green build is *not* sufficient evidence that the registration is live. `CreateStaticPluginEpLibraries()` in -`onnxruntime/core/session/plugin_ep/ep_static_plugins.cc` is guarded by -`#if defined(USE_WEBGPU) && defined(ORT_WEBGPU_STATIC_PLUGIN)`; if that define does not reach ORT core the function -still compiles, simply returning an empty vector, and the build is just as green while registering nothing. The -check that actually discriminates is at the symbol level — ORT core must hold an *undefined* reference to the -prefixed entry points, resolved by the provider library: +## Validation summary -``` -llvm-nm libonnxruntime_session.a | grep WebGpu_ # U WebGpu_CreateEpFactories / U WebGpu_ReleaseEpFactory -llvm-nm libonnxruntime_providers_webgpu.a | grep WebGpu_ # T WebGpu_CreateEpFactories / T WebGpu_ReleaseEpFactory -``` +The static plugin configuration builds and runs on Windows with a real GPU, and its test results are identical to +the internal-EP build: across the 5894 `onnxruntime_provider_test` cases common to both, zero status differences. +Linux GCC and Emscripten are covered by build-only CI legs. The known gaps are operator coverage on the plugin path +for internal-only white-box tests, environment re-creation, and any runtime execution of the Emscripten build. -Both were observed on the Emscripten build, which confirms the guard was live, the D3 entry-point prefixing worked, -and the link resolved core's reference against the provider. That check validates build, link and registration -wiring; runtime behaviour was established separately by the ORT Web prototype described under -[ORT Web migration](#ort-web-migration). - -A practical build note: the WASM build needs `node` on `PATH` — `node_helper.cmake` does a hard `find_program` and -fails configure without it. emsdk bundles a suitable one under `cmake/external/emsdk/node//bin`. - -### Native shared-library plugin performance - -The latency criterion covers two shapes of plugin, and the ORT Web numbers under -[Measured performance comparison](#measured-performance-comparison) settle only the static one. A -native `--use_webgpu shared_lib` A/B against a built-in baseline measured the plugin slower by ~8% on a -dispatch-bound model and 4-6% on Qwen3.5-0.8B, fitting roughly **67 µs per `Run` plus 1.2 µs per node**. The -reproducibility caveat recorded for the web numbers applies here too; these figures should be re-taken with the -commit, flags and harness recorded. - -One control is what licenses reading the delta as boundary cost at all: matching build flags do not prove two -executables perform alike, so both binaries were also run on the CPU EP, where no plugin is loaded. That gave a -ratio of minima of 1.001 — but only single-threaded. Run with default threading the same control showed the -plugin executable 24% *faster*, which would have been actively misleading. - -**Root cause: redundant per-kernel work, not boundary crossing.** 1.2 µs/node is ~4,400 cycles, far more than a -cross-DLL indirect call can account for, so both arms were CPU-profiled. `ep::adapter::CreateTensorFromApiValue` -rebuilds an `onnxruntime::Tensor` — nine C API round trips, an allocator-name `std::string`, a `TensorShape` -allocation — for every input and output of every node on every `Run`, because the adapter's `OpKernelContext` is -constructed fresh per `Compute()` and caches nothing beyond it. The adapter symbols are absent from the built-in -profile entirely, `Tensor`/`TensorShape` work doubles, and the extra CPU (+0.53 ms/rep) accounts for the entire -extra wall latency (+0.42 ms/rep). - -That makes the cost look **reducible rather than intrinsic**, which matters for how the latency criterion is -settled: negotiating a tolerance around a fixable defect would be the wrong order. The obvious candidates are -caching the `Tensor` wrapper across `Compute()` calls, and avoiding the allocator-name `std::string` and the -`TensorShape` allocation on the per-node path. This is also the leading candidate for the unattributed web -per-node cost, since both boundaries charge per kernel node. - -Two gaps remain in the native numbers. Session-creation cost was not compared — the plugin arm must also discover -and load the DLL, so it is likely worse in relative terms. And both arms ran with default WebGPU options; graph -capture in particular targets exactly the per-dispatch cost identified above and is worth a follow-up. - -**The plugin path silently loses the WebGPU EP's `Api` profiling events.** The built-in arm emits 15,300 `Api` -category events under `-p` and the plugin arm emits none. This is an observability regression independent of -latency, and it also means ORT's own profiler cannot perform this A/B: under `-p` the built-in arm measures -*slower* and the result inverts. - -## ORT Web migration - -The intended end state is that the WebGPU EP is always a plugin EP and `--use_webgpu static_lib` goes away -entirely, ORT Web included. Building the EP into the WASM binary as a static plugin, which this change enables, is -only the first half of that. The second half is *selection*, and two gaps stand between the current state and a -working ORT Web on the plugin path. Neither is visible to the linker, so the symbol-level evidence above does not -speak to either. - -Both gaps have since been closed by a throwaway prototype, which ran the full `js/web` WebGPU operator suite in -Edge against a WASM `static_plugin` binary: 2152 tests, all passing. The design below is therefore validated end -to end rather than merely plausible; what remains is productionization, not discovery. The prototype's findings are -recorded inline in the steps that follow. - -**ORT Web selects the EP by name, and that name resolves only to the internal EP.** -`js/web/lib/wasm/session-options.ts` maps the `'webgpu'` backend to `epName = 'WebGPU'`, which reaches -`OrtAppendExecutionProvider` in `onnxruntime/wasm/api.cc` and from there -`OrtApis::SessionOptionsAppendExecutionProvider`. That API is a closed table of hardcoded EP names in -`onnxruntime/core/session/provider_registration.cc`, and its `EpID::WebGPU` case is compiled out under -`ORT_USE_EP_API_ADAPTERS`. On a `static_plugin` build the EP therefore registers at `OrtEnv` creation and is then -unselectable from JS: the call returns "WebGPU execution provider is not supported in this build". This is by -design rather than an oversight — the name-based API is the internal-EP mechanism, and plugin EPs are selected -through `RegisterExecutionProviderLibrary` plus `OrtEpDevice`-based selection — but it does mean ORT Web needs new -plumbing rather than a recompile. - -**No WebGPU `OrtEpDevice` exists under Emscripten.** Emscripten was neither `WIN32`, `LINUX` nor `APPLE`, so -`cmake/onnxruntime_common.cmake` fell through to `core/platform/device_discovery_default.cc`, which discovers only -a CPU device. `Factory::GetSupportedDevicesImpl` in `onnxruntime/core/providers/webgpu/ep/factory.cc` creates an -`OrtEpDevice` only for devices of type `OrtHardwareDeviceType_GPU`. The one fallback, the virtual GPU device in the -same function, is opt-in through the `allow_virtual_devices` environment configuration entry and is deliberately -registered without allocator info, because it exists to back a device-free compile-only session. So even after -selection plumbing is added, `GetEpDevices` would offer nothing for JS to select. - -A workable order: - -1. Make WebGPU visible as a device under Emscripten by adding an Emscripten device discovery implementation. - `cmake/onnxruntime_common.cmake` gains an `Emscripten` arm selecting a new - `core/platform/emscripten/device_discovery.cc`, which reports the CPU device via `GetCpuDeviceFromCPUIDInfo()` - plus a GPU `OrtHardwareDevice` when a synchronous `EM_ASM_INT` check finds `navigator.gpu`. No change to - `Factory::GetSupportedDevicesImpl` is needed: its existing loop filters on `OrtHardwareDeviceType_GPU`, so a - discovered GPU flows through and picks up both allocator infos unmodified. - - The prototype first tried the other option — having the factory synthesize the device itself through - `CreateHardwareDevice` — and it worked, but it is the wrong layer. `GetSupportedDevices` documents its `devices` - argument as "the `OrtHardwareDevice` instances that are available", so that array is the source of truth, and - `CreateHardwareDevice` is documented for devices with no hardware behind them ("e.g., virtual"). A browser GPU - is real hardware that ORT's discovery layer merely failed to enumerate, so the producer is the place to fix it. - Leaving it in the factory would also oblige every future Emscripten plugin EP to repeat the same workaround. - - A capability check is a legitimate form of discovery here. `core/platform/apple/device_discovery.cc` sets the - precedent: it enumerates nothing, it asserts that one GPU exists on Apple Silicon and hardcodes the vendor id. - `navigator.gpu` is a stronger signal than that — an actual runtime capability probe, and the same one ORT Web's - `initEp` uses to decide whether WebGPU is usable. `EM_ASM` from ORT core is likewise established practice - (`core/graph/model.cc`, `core/framework/external_data_loader.cc`), not just an EP-layer habit. - - The honest limitation is that no adapter information is available. `_OrtInit` creates the `OrtEnv` and therefore - triggers discovery, and it runs before `navigator.gpu.requestAdapter()` in ORT Web's `initEp`, so `vendor_id` - and `device_id` are left at 0. That is acceptable: Apple reports no device id either, and browsers mask - `GPUAdapterInfo.vendor` for fingerprinting resistance regardless. The device is deliberately *not* tagged with - `kOrtHardwareDevice_MetadataKey_IsVirtual`, which is what keeps it distinct from the virtual device — that path - is allocator-less by design and cannot back a real session. - - A later change on `main` added a second fallback to `GetSupportedDevicesImpl` alongside the virtual device: when - no GPU device is discovered, `allow_software_adapter` (opt-in via the `kAllowSoftwareAdapterEnvironmentVariable` - environment variable) advertises WebGPU against the *CPU* `OrtHardwareDevice`. It does not change the analysis - above. Both upstream fallbacks work around a device the discovery layer failed to enumerate; they are opt-in - precisely because misreporting a device is not safe to do by default. Under Emscripten the device is genuinely - there and cheaply detectable, so fixing discovery keeps the default path correct and needs no opt-in. The - primary matching loop that our discovered GPU device flows through is unchanged by that work. - - One accepted side effect: `ep_library_internal.cc` creates an internal `WebGpuEpFactory` in a - `USE_WEBGPU && !ORT_USE_EP_API_ADAPTERS` build, so today's `static_lib` ORT Web build starts producing an - `OrtEpDevice` where it previously produced none. That build selects by name and is slated for removal, so the - change is benign. - -2. Expose plugin EP selection in the WASM C API. `g_env` is already available in `api.cc`, so `GetEpDevices` and - `SessionOptionsAppendExecutionProvider_V2` are both reachable. A narrow helper that appends by EP name and - resolves the device internally avoids marshalling arrays of device pointers across the JS boundary. The new - entry point needs to be exported from `cmake/onnxruntime_webassembly.cmake`. - - *Prototype result:* the name-resolving helper is the right shape. Note that exporting it takes three edits, not - one: `EMSCRIPTEN_KEEPALIVE` in `api.h`, the `JSPI_EXPORTS` list in `cmake/onnxruntime_webassembly.cmake` for - JSPI builds, and the `wrapAsyncAPIs` list in `onnxruntime/wasm/pre-async.js` for Asyncify builds. Omitting the - last one is the easy mistake, and Asyncify is the default variant. Reporting matters too: on no match the helper - should raise a status naming the EPs that *are* registered, which is what makes a misconfiguration diagnosable - from JS through `checkLastError`. - -3. Switch `js/web` to the new call. The `BUILD_DEFS.DISABLE_WEBGPU` branch that falls back to the JS EP is - unaffected. Confirm that the EP options ORT Web passes today — `deviceId`, `webgpuInstance`, `webgpuDevice`, - `preferredLayout` and the buffer cache modes — still reach the provider, since `Factory::CreateEpImpl` obtains - them from the session config options. - - *Prototype result:* the options do reach the provider unchanged, and the reason is worth recording. Both paths - derive their config prefix from `OrtSessionOptions::GetProviderOptionPrefix` keyed on the EP name, so V2 lands - the options under exactly the same `ep.webgpuexecutionprovider.*` keys the name-based path used. One JS change - is required though: `session-options.ts` passes the short name `'WebGPU'`, whereas `OrtEpDevice` selection - matches the canonical `'WebGpuExecutionProvider'` returned by `EpDevice_EpName`. - - The productionization question this raises is *conditionality*: is it safe to switch to V2 unconditionally - whenever WebGPU is enabled? It is, provided the WASM build flips at the same time. `BUILD_DEFS.DISABLE_WEBGPU` - is defined in `js/web/script/build.ts` as exactly `!USE_WEBGPU_EP`, so the bundle is already specialized per - WASM flavour by `--webgpu-ep` — the JSEP bundle takes the other branch entirely and is unaffected. Flipping the - WebGPU WASM builds to `static_plugin` (step 4) therefore makes `!DISABLE_WEBGPU` imply `static_plugin` by - construction, and no additional build define or runtime capability probe is needed. What is *not* safe is - landing the JS change without the build change, since the two must move together. - -4. Flip the ORT Web WASM builds to `static_plugin` and extend CI to *test* the result. The two builds are - `Build (simd + threads + WebGPU experimental)` and its JSPI sibling in - `.github/workflows/linux-wasm-ci-build-and-test-workflow.yml`, both of which pass a bare `--use_webgpu` and so - get `static_lib` from `build_args.py`'s `const="static_lib"`. Neither uses `--minimal_build`, so the - minimal-build `FATAL_ERROR` guard does not stand in the way of this migration. A build-only `static_plugin` - leg already runs (see [Emscripten and ORT Web](#emscripten-and-ort-web)), so this path is covered at compile - time; flipping these two builds is what gets it *executed*, because the published artifacts then become the - plugin build and the existing `web_Debug` / `web_Release` suites cover it with no parallel test lane. - - *Prototype note:* the `js/web` test runner cannot currently exercise this configuration end to end, because - `script/test-runner-cli.ts` spawns `script/build` with only `--bundle-mode` and does not forward `--webgpu-ep`. - Setting `npm_config_webgpu_ep` in the environment works as a stopgap, since `script/build.ts` reads it, but a CI - leg should forward the flag properly. Note also that without `--webgpu-ep` the `webgpu` backend silently routes - through JSEP instead, so a leg that omits it would pass while testing nothing relevant. - -### Interaction with the ORT Web reduced-size build options - -The ORT Web WASM builds are size-sensitive and pass a set of size-reduction flags that the local prototype builds -did not. `linux-wasm-ci-build-and-test-workflow.yml` applies `--disable_ml_ops --disable_generation_ops ---disable_types string float4 float8 optional sparsetensor --include_ops_by_config -onnxruntime/wasm/reduced_types.config --enable_reduced_operator_type_support`, and `web.yml` adds `--disable_rtti` -for the release job. Since the migration changes how the WebGPU EP is compiled and linked, each of these was -checked against the plugin path. - -**None of them need adapting.** The findings: - -- *Compile definitions reach the WebGPU target in every mode.* `onnxruntime_providers_webgpu` is created through - `onnxruntime_add_static_library` / `onnxruntime_add_shared_library_module`, both of which call - `onnxruntime_configure_target` → `onnxruntime_set_compile_flags`. That function is where every `DISABLE_*` and - `REDUCED_OPS_BUILD` definition is applied, so the `static_lib`, `static_plugin` and shared-library builds all - receive an identical set. There is no propagation gap introduced by the plugin boundary. - -- *Operator and type reduction never applied to WebGPU, before or after.* - `op_registration_utils.get_kernel_registration_files()` hardcodes the CPU registration files (plus CUDA when - requested); WebGPU is not in the list, so `--include_ops_by_config` does not rewrite WebGPU kernel - registrations. `onnxruntime/wasm/reduced_types.config` is `!no_ops_specified_means_all_ops_are_required` — it - performs global *type* reduction only, with no op exclusion — and that mechanism works through the - `op_kernel_type_control` macros, which neither `core/providers/webgpu` nor `contrib_ops/webgpu` uses. Its saving - is entirely CPU-EP-side and is unchanged by the plugin switch. This is a pre-existing gap, not a regression. - -- *The type and ML/generation op flags are inert for WebGPU.* No source under `core/providers/webgpu` or - `contrib_ops/webgpu` references `DISABLE_ML_OPS`, `DISABLE_GENERATION_OPS`, `DISABLE_SPARSE_TENSORS`, - `DISABLE_OPTIONAL_TYPE`, `DISABLE_FLOAT8_TYPES`, `DISABLE_FLOAT4_TYPES` or `DISABLE_STRING_TYPE`. - -- *`--disable_rtti` is safe on the plugin path.* Neither `include/onnxruntime/ep` (the EP API adapters) nor - `onnxruntime/core/session/plugin_ep` uses `dynamic_cast` or `typeid`. - -One consequence is worth recording for the *shared library* plugin EP, which is out of scope here but shares this -code. `onnxruntime_c_api.h` contains no `DISABLE_*` guards, so the `OrtApi` struct layout is invariant under these -flags and the ABI is stable. `onnxruntime_cxx_api.h` does guard some `Ort::Value` members (the sparse tensor -methods) — an EP DLL built with a different `--disable_types` than its host would therefore see a different C++ -header surface, but since those wrappers are inline over stable function pointers this is a source-compatibility -concern rather than an ABI one. For `static_plugin` everything is compiled in one tree with one set of defines, so -the question does not arise. - -**What is *not* covered by any existing flag** is the plugin machinery itself. Relative to `static_lib`, the -`static_plugin` build adds `core/providers/webgpu/ep/*` (excluded from the source list in the `static_lib` branch -of `onnxruntime_providers_webgpu.cmake`), inlines the header-only EP API adapters into every WebGPU translation -unit, and reaches the `OrtGraph` / `OrtNode` C API graph views through `OrtEp::GetCapability`. Note that -`core/session/plugin_ep` is *not* part of that delta: `onnxruntime_session.cmake` only excludes it for -`onnxruntime_MINIMAL_BUILD`, so today's WASM builds already compile it in. - -That also means the only existing lever that trims plugin-EP machinery is `--minimal_build`, which -`static_plugin` currently rejects outright. If measurement shows the size delta matters, lifting that restriction -(already listed as a follow-up) is the mechanism to reach parity — not a new size-reduction flag. - -#### Measured size comparison - -The ORT Web npm package ships **two** WebGPU WASM artifacts, not one, and both were measured. The release path -(`npm-packaging-pipeline.yml` → `templates/web-ci.yml` → `templates/linux-wasm-ci.yml`) and the GitHub Actions path -(`web.yml` → `linux-wasm-ci-build-and-test-workflow.yml`) pass an identical flag set for the WebGPU legs, which -differ from each other only in their exception/stack-switching mode: - -| Leg | Artifact | Distinguishing flag | -| --- | --- | --- | -| `wasm_inferencing_webgpu` | `ort-wasm-simd-threaded.asyncify.{wasm,mjs}` | `--enable_wasm_api_exception_catching` | -| `wasm_inferencing_webgpu_jspi` | `ort-wasm-simd-threaded.jspi.{wasm,mjs}` | `--enable_wasm_jspi` | - -Four clean from-scratch builds were run (both EP modes × both legs) with the exact release flag set: `Release`, -SIMD + threads, WebNN on, `--disable_rtti`, `--target onnxruntime_webassembly`, and the full reduced-size set -above. Compressed sizes are included because that is what is actually delivered over the wire; they were produced -with gzip level 9 and brotli quality 11. - -`ort-wasm-simd-threaded.asyncify.wasm`: - -| Metric | `static_lib` | `static_plugin` | Delta | -| --- | ---: | ---: | ---: | -| raw | 26,549,423 B | 26,487,397 B | **−62,026 B (−0.234%)** | -| gzip | 6,537,701 B | 6,566,088 B | **+28,387 B (+0.434%)** | -| brotli | 3,953,625 B | 3,971,550 B | **+17,925 B (+0.453%)** | - -`ort-wasm-simd-threaded.jspi.wasm`: - -| Metric | `static_lib` | `static_plugin` | Delta | -| --- | ---: | ---: | ---: | -| raw | 16,627,577 B | 16,687,113 B | **+59,536 B (+0.358%)** | -| gzip | 4,099,914 B | 4,126,147 B | **+26,233 B (+0.640%)** | -| brotli | 2,651,643 B | 2,674,520 B | **+22,877 B (+0.863%)** | - -The accompanying `.mjs` files are byte-identical between the two EP modes in both legs (53,300 B asyncify, 51,204 B -JSPI); their compressed sizes differ by at most 23 B. - -**Every delta is under 1%, which is the substantive conclusion.** Two details are worth recording so the numbers -are not over-read: - -- *The raw sign is not stable across the two artifacts.* `static_plugin` is marginally smaller raw on asyncify but - marginally larger raw on JSPI. An earlier revision of this document reported only the asyncify raw number and - generalised it to "no size regression"; that generalisation was too broad. The asyncify measurement itself - reproduces well — it was −61,968 B when first taken and −62,026 B in this from-scratch rerun at a later commit. - -- *Compressed size grows slightly in both legs*, by +0.4% to +0.9%, so the direction is consistent once the - artifacts are compressed. Why compressed size can rise while raw size falls was not investigated; no - symbol-level or per-section attribution was done, and none of the earlier speculation about *which* code the - plugin build drops or adds has been measured. Given the magnitude, this was not pursued further. - -Both configurations compiled cleanly with every reduced-size flag in both legs, which empirically confirms the -compatibility analysis above. For reference, the same `static_plugin` asyncify configuration without any -reduced-size flags is 40.9 MB, so those flags are worth ~15 MB and remain essential regardless of EP mode. - -#### `--enable_wasm_api_exception_catching` and the EP API adapters - -The one option that *did* interact with the plugin path is `--enable_wasm_api_exception_catching`. It is not a -size-reduction flag in the same sense as the others, but it is part of the same CI flag set and it exposed a real -bug. - -`onnxruntime_webassembly.cmake` compiles only `wasm/api.cc` and `core/session/onnxruntime_c_api.cc` with -`-sDISABLE_EXCEPTION_CATCHING=0`; every other translation unit keeps Emscripten's default, where `catch` clauses -are compiled so that they never match. Exceptions still propagate — they are simply only catchable at the C API -boundary. +Full results, CI job inventory, symbol-level checks and native performance measurements are in +[Static Plugin EP Registration: Validation](static_plugin_ep_registration_validation.md). -The EP API adapters in `include/onnxruntime/ep/adapter/` are header-only and are therefore inlined into WebGPU -translation units that have catching disabled. `OpKernelInfo::GetAttr()` / `GetAttrs()` were implemented as a -`try` around the throwing `Ort::` C++ wrappers, converting `Ort::Exception` into a `Status`. That makes a *missing -optional attribute* — ordinary control flow for `GetAttrOrDefault()` — depend on catching an exception. With -catching disabled the `catch` never ran, so the exception escaped `GetAttrOrDefault()` entirely and surfaced from -session creation as e.g. `ERROR_CODE: 6, ERROR_MESSAGE: No attribute with name:'extrapolation_value'is defined.` - -This reproduced as 949 `suite0` failures on the reduced-size `static_plugin` build while the same build passed on -the CPU backend, the reduced-size `static_lib` build passed, and the full-op `static_plugin` build passed — the -failure required plugin + WebGPU + API-only exception catching together. - -The fix is to not use exceptions for control flow in the adapters: `GetAttr()` / `GetAttrs()` now call the -non-throwing `OrtApi::KernelInfoGetAttribute*` functions directly and convert the returned `OrtStatus*` into a -`Status`. **General rule: header-only EP API adapter code must not rely on catching exceptions**, because a plugin -EP may be compiled with exception catching disabled. The remaining `catch` blocks in `include/onnxruntime/ep` -(`common.h`, `get_capability_utils.h`, `adapter/kernel_registry.h`) sit on genuinely exceptional callback -boundaries rather than on control-flow paths; when catching is disabled those exceptions still reach the outer C -API boundary and are reported, only with a less specific message. - -With that fix the reduced-size `static_plugin` build passes `suite0` on the WebGPU backend in full: 2152 tests, -all passing, matching the `static_lib` control run on the identical flag set. - -#### Measured performance comparison - -The same two binaries were compared on inference performance in Edge, using the test runner's `--perf` mode -(`test-runner-cli model -b=webgpu -e=edge -P=`) against two synthetic models chosen to isolate the two -places the plugin path could plausibly cost something: - -- **dispatch-bound** — 300 chained tiny elementwise ops (`Mul`/`Add`/`Sub`) on a `[1, 1024]` tensor. Almost no GPU - work per node, so the run time is dominated by per-node host-side work, which is exactly what now goes through - the C API adapters. -- **compute-bound** — 16 chained `[512, 512]` `MatMul`s. Host-side per-node cost is negligible against the GPU - work. - -Three runs per model per build. **The two builds must be sampled interleaved** — one round is -`plugin/dispatch, plugin/compute, lib/dispatch, lib/compute`, repeated — rather than all samples of one build -followed by all samples of the other. This machine drifts: a build that has just finished, a warm browser, or a -warm GPU shifts timings by more than the effect being measured, and sequential sampling aliases that drift onto -the build under test. The table reports the median of the three per-run P50 values. - -| Metric | `static_lib` | `static_plugin` | Delta | -| --- | ---: | ---: | ---: | -| dispatch-bound, per-run P50 | 11.20 ms | 12.00 ms | +0.8 ms (+7%) | -| compute-bound, per-run P50 | 7.70 ms | 7.80 ms | +0.1 ms (within noise) | - -Reading of these numbers: - -- **Execution is not measurably slower once GPU work dominates.** The compute-bound samples overlap - (`static_plugin` 7.80 / 7.80 / 7.40 against `static_lib` 7.90 / 7.00 / 7.70 — the plugin build's best sample - beats two of the three control samples), so the difference is not resolvable at this sample count. -- **A dispatch-bound graph pays roughly 7%.** That is the cost of routing per-node kernel work through the C API - adapters instead of a direct in-process virtual call. Unlike the compute-bound case this one does separate - cleanly — the plugin build's slowest control sample is still faster than its own fastest sample. It only shows - up when nodes do essentially no GPU work, which is the worst case by construction rather than a representative - one. Amortized over ~300 nodes it is on the order of 3 µs per node — but see the caveat below before relying on - that figure. -- **Session creation costs about 3%** (measured pre-merge: 1136 → 1172 ms and 1250 → 1290 ms for the two models). - This covers plugin EP registration, `GetCapability` and kernel creation through the C API. The absolute figure - is dominated by fetching and parsing the model, so the true relative cost of the plugin machinery within session - creation is higher than 3% — but it is tens of milliseconds once, not per inference. - -Two cautions on methodology: - -- The *total* wall time the test runner prints for `suite0` is not a usable metric. It varied between 3 min 15 s - and 4 min 04 s for the **same** `static_lib` binary across two runs. Only the second figure it prints — net test - time, excluding the per-model `before all` session-creation hooks — is stable enough to compare, and by that - measure the two builds are equal. An early single-sample comparison of the total figure suggested a 29% - regression that does not exist. -- Timings taken with `--webgpu.profiling.mode=default` are inflated by the timestamp queries and must never be - compared against timings taken without it. - -**The per-node magnitude is not reliably established, and a controlled re-measurement is outstanding.** Three -separate measurements of the same 300-node dispatch-bound model disagree by roughly 3.7x: +0.80 ms (2.7 µs/node) -above, +1.50 ms (5.0 µs/node) in a later four-model sweep that fitted 4.76 µs/node across 150/300/600 nodes, and -+3.00 ms (10.0 µs/node) in a four-side build comparison. The `static_lib` baseline itself moved from 7.90 ms to -11.90 ms for the same model on the same machine within a day, so build-to-build and session drift dominates the -spread, and Edge's 0.1 ms `performance.now()` floor without cross-origin isolation limits resolution further. The -three runs also did not use the same statistic (per-run P50 versus median-of-round-minimum). A re-measurement -should record the commit of each arm, the full build flag set, the model definitions, the statistic, the -interleaved sampling order and the GPU submit counts, so the number can be reproduced. - -What *is* robust is the sign and the mechanism. The plugin side was slower in 26 of 26 paired rounds in the sweep -and 6 of 6 in a three-way run, and GPU submit counts were byte-identical on every side, model and round, so the -cost is CPU-side per-kernel work rather than extra queue work. The same three-way run also excludes the -JavaScript side: a `mismatch` arm (plugin bundle against a `static_lib` wasm) came out level with `static_lib`, so -the `_OrtAppendExecutionProviderV2` call and the EP-name lookup are free and the cost is inside the -`static_plugin` wasm's per-kernel C++ path. **Where exactly is not yet attributed** — an `--enable_wasm_profiling` -build would give named frames, and the native root cause recorded under -[Native shared-library plugin performance](#native-shared-library-plugin-performance) is the leading candidate. +ORT Web's migration onto this path, including the Emscripten build status, the remaining migration steps and the +size and performance analysis, is in +[ORT Web Static Plugin Migration](ort_web_static_plugin_migration.md). diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_validation.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_validation.md new file mode 100644 index 0000000000000..c87fba6eb6e19 --- /dev/null +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_validation.md @@ -0,0 +1,138 @@ +# Static Plugin EP Registration: Validation + +Evidence for [Static Plugin EP Registration](static_plugin_ep_registration_design.md). That document defines the +design and its contracts; this one records what was built, what was run, and what is still untested. + +## Test results + +Verified on Windows with `--use_webgpu static_plugin`, Debug: + +- `onnxruntime_test_all` and `onnxruntime_provider_test` build and link. +- The statically linked WebGPU plugin EP is registered by ORT core during `Ort::Env` creation, and its `OrtEpDevice` + is enumerated and selectable, confirming the D6 publish-then-register ordering resolves the self-deadlock. +- `InferenceSessionTests.WebGpuVirtualDeviceCompileOnlyEndToEnd` and + `InferenceSessionTests.WebGpuVirtualDeviceRejectedWithoutCompileOnly` pass, which covers session creation and EP + instantiation end to end through the public V2 API on the static plugin path. + +Operator execution was initially not verifiable locally: the original development machine was a virtual machine with +no real GPU, so only the virtual WebGPU device was surfaced and non-compile-only sessions were rejected up front. This +was confirmed to be a property of the host rather than of static linking by building the shared library plugin +configuration (`--use_webgpu shared_lib`) on the same host and running the same tests: it passed and failed exactly +the same set (`WebGpuVirtualDeviceCompileOnlyEndToEnd` and `WebGpuVirtualDeviceRejectedWithoutCompileOnly` pass; +`WebGpuCompileOnlySkipsFinalization` and `TestStrictShapeInference` fail with the identical virtual-device rejection +message). + +This gap has since been closed on a machine with a real GPU (NVIDIA RTX 5060 Ti). The `static_plugin` and internal-EP +configurations were built from the same tree, RelWithDebInfo, differing only in the EP path, and their test results +compared: + +- `onnxruntime_provider_test`: 0 failures in both. **Across the 5894 tests common to both builds there were zero + status differences.** +- `onnxruntime_test_all`: 1965 passed in both, no differences. All three `InferenceSessionTests.WebGpu*` pass on the + static plugin build, including `WebGpuCompileOnlySkipsFinalization`, confirming its earlier failure was indeed a + property of the GPU-less host. + +## Known test coverage difference on the plugin path + +`cmake/onnxruntime_unittests.cmake` gates the `test/providers/webgpu/*` sources on +`onnxruntime_USE_WEBGPU AND NOT onnxruntime_USE_EP_API_ADAPTERS`, so those tests are **absent** from any adapter +build, including `static_plugin`. They are white-box tests that include internal WebGPU EP headers and therefore +cannot compile against the adapter boundary by construction. + +Of the 44 tests this excludes, 16 are `DISABLED_` and never run, and most of the rest assert on EP internals +(`WebGpuContextTest`, `ActivationCacheKeyTest`). However, 12 of them — `HardSwish_WebGPU` and `MatMul2BitsWebGpu` — +are genuine operator tests, and they currently run **only** on the internal EP path. This is a known coverage gap of +the plugin path. Closing it would mean rewriting them as EP-agnostic operator tests so both paths execute them; that +is deliberately left out of this change to keep it focused. + +Separately, the `InferenceSessionTests.WebGpu*` virtual-device tests are deliberately *paired* rather than shared +between the two builds, since each drives a different factory. `WebGpuVirtualDeviceCompileOnlyEndToEnd` and +`WebGpuVirtualDeviceRejectedWithoutCompileOnly` are the plugin-build counterparts of `WebGpuEpFactoryVirtualDevice` +and `WebGpuEpFactoryRejectsVirtualDeviceWithoutCompileOnly`. Only `WebGpuCompileOnlyUsesNoOpAllocator` has no +counterpart, because it asserts an internal allocator *type* through `dynamic_cast`, which is inherently +unobservable across the ABI boundary. + +The shared library plugin configuration was also re-verified after the D7 change, since `include/onnxruntime/ep/api.h` +is shared with the CUDA plugin EP. It builds, and `onnxruntime_providers_webgpu.dll` still exports exactly the +unprefixed `CreateEpFactories` and `ReleaseEpFactory` required by `EpLibraryPlugin::Load`, per D2. + +## CI coverage + +No pre-existing CI leg built this configuration, since both new CMake options default to `OFF`. Three legs were +added: + +- `webgpu_static_plugin_build_x64_RelWithDebInfo` in `.github/workflows/windows_webgpu.yml`, which builds and runs the + tests on a GPU-equipped pool. This closes the operator execution gap above. Note that it must not set + `onnxruntime_BUILD_DAWN_SHARED_LIBRARY=ON`, which is incompatible with `onnxruntime_USE_EP_API_ADAPTERS`. +- `build-linux-webgpu-static-plugin-x64-release` in `.github/workflows/linux_webgpu.yml`, build only, mirroring its + sibling job. This is the only GCC coverage of the EP adapters, and the only coverage of the `onnxruntime_python.cmake` + change, hence `--build_wheel` is kept. +- A build-only `--use_webgpu static_plugin` step in the `wasm_Release` job of `.github/workflows/web.yml`, which + keeps the Emscripten registration path compiling. It publishes no artifacts; see + [ORT Web Static Plugin Migration](ort_web_static_plugin_migration.md#emscripten-and-ort-web) for why, and for what + that leg does and does not prove. + +The first run of the Linux job surfaced `-Werror=maybe-uninitialized` errors. The EP adapter `OpKernelInfo` defines its +attribute accessors inline in the header, so GCC can see their failure and exception paths, whereas the in-tree +`OpKernelInfo` hides them behind an out-of-line definition. Any value that is only read when the attribute lookup +succeeded must be value-initialized. + +More generally, this job is the first time the EP adapters have ever been compiled with GCC, and it has surfaced +several successive batches of `-Werror` diagnostics — including false positives inside `absl::InlinedVector` and, in +optimized builds, `-Werror=array-bounds` on `std::string` concatenation chains that the adapter headers cause to be +inlined more aggressively. Because ninja stops at the first failure and reports only the targets already in flight, +**each CI run reveals just one batch**. When touching this configuration, enumerate the full set locally in a single +pass — an AlmaLinux 8 container with `gcc-toolset-14` reproduces the CI compiler exactly — rather than discovering +them one CI round trip at a time. Fixes that are correct on their own merits are applied at the source; irreducible +false positives use a suppression scoped to the `onnxruntime_providers_webgpu` target, GCC only, and conditional on +`onnxruntime_USE_EP_API_ADAPTERS`, so non-adapter builds stay strict. This mirrors what the CUDA plugin EP does in +`cmake/onnxruntime_providers_cuda_plugin.cmake`. + +That full local enumeration has since been done, and it is worth recording what it found: of the ~340 warnings GCC 14 +emits across the whole `static_plugin` build, **exactly one is attributable to ORT's own source** — the +`-Warray-bounds` false positive in `contrib_ops/webgpu/quantization/matmul_nbits_mlp.cc`, fixed by building the +expression with `MakeString` instead of a `std::string` concat chain. Everything else lives in Dawn, ONNX, +flatbuffers, or the telemetry dependency, in targets that do not use `-Werror`. The practical lesson is that the +adapters do not make ORT's code broadly warning-prone; they perturb inlining at a small number of specific sites, +so a targeted source fix is almost always available and preferable to a blanket target-wide suppression. + +When validating such a fix locally, always rebuild the *unfixed* source through the same harness first and confirm it +still errors. A harness that silently fails to exercise the warning is indistinguishable from a successful fix. + + +## Native shared-library plugin performance + +The latency criterion covers two shapes of plugin, and the ORT Web numbers under [Measured performance +comparison](ort_web_static_plugin_migration.md#measured-performance-comparison) settle only the static one. A +native `--use_webgpu shared_lib` A/B against a built-in baseline measured the plugin slower by ~8% on a +dispatch-bound model and 4-6% on Qwen3.5-0.8B, fitting roughly **67 µs per `Run` plus 1.2 µs per node**. The +reproducibility caveat recorded for the web numbers applies here too; these figures should be re-taken with the +commit, flags and harness recorded. + +One control is what licenses reading the delta as boundary cost at all: matching build flags do not prove two +executables perform alike, so both binaries were also run on the CPU EP, where no plugin is loaded. That gave a +ratio of minima of 1.001 — but only single-threaded. Run with default threading the same control showed the +plugin executable 24% *faster*, which would have been actively misleading. + +**Root cause: redundant per-kernel work, not boundary crossing.** 1.2 µs/node is ~4,400 cycles, far more than a +cross-DLL indirect call can account for, so both arms were CPU-profiled. `ep::adapter::CreateTensorFromApiValue` +rebuilds an `onnxruntime::Tensor` — nine C API round trips, an allocator-name `std::string`, a `TensorShape` +allocation — for every input and output of every node on every `Run`, because the adapter's `OpKernelContext` is +constructed fresh per `Compute()` and caches nothing beyond it. The adapter symbols are absent from the built-in +profile entirely, `Tensor`/`TensorShape` work doubles, and the extra CPU (+0.53 ms/rep) accounts for the entire +extra wall latency (+0.42 ms/rep). + +That makes the cost look **reducible rather than intrinsic**, which matters for how the latency criterion is +settled: negotiating a tolerance around a fixable defect would be the wrong order. The obvious candidates are +caching the `Tensor` wrapper across `Compute()` calls, and avoiding the allocator-name `std::string` and the +`TensorShape` allocation on the per-node path. This is also the leading candidate for the unattributed web +per-node cost, since both boundaries charge per kernel node. + +Two gaps remain in the native numbers. Session-creation cost was not compared — the plugin arm must also discover +and load the DLL, so it is likely worse in relative terms. And both arms ran with default WebGPU options; graph +capture in particular targets exactly the per-dispatch cost identified above and is worth a follow-up. + +**The plugin path silently loses the WebGPU EP's `Api` profiling events.** The built-in arm emits 15,300 `Api` +category events under `-p` and the plugin arm emits none. This is an observability regression independent of +latency, and it also means ORT's own profiler cannot perform this A/B: under `-p` the built-in arm measures +*slower* and the result inverts. From 1bae7834dd72295e9152a95c9b734afaf6f8dcff Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Fri, 11 Sep 2026 12:29:15 -0700 Subject: [PATCH 074/107] Record per-package status in the plugin boundary workstream The workstream listed seven work packages with no indication of what had been done, so a reader could not tell the static-registration milestone apart from the end state. Give each package an explicit status with the evidence behind it. Package 4 (Emscripten prototype) is done: a throwaway prototype ran the full js/web WebGPU operator suite in Edge against a WASM static_plugin binary, 2152 tests passing. Packages 2, 3 and 7 are in progress, each with the specific remaining gap named rather than left implicit: entry-point prefix uniqueness is unexercised while WebGPU is the only static plugin, the process-global inventory and the re-registration regression are outstanding, and legacy-path removal waits on ORT Web parity. Package 1 is in progress because the latency numbers exist but are not reproducible. Packages 5 and 6 are not started. Also note against the completion criteria that none are met yet, since they describe the whole workstream rather than this milestone, and replace the "Working plan" document status with what has actually landed. --- ...boundary_and_web_integration_workstream.md | 53 +++++++++++++++---- 1 file changed, 43 insertions(+), 10 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md index 8efe3e0b0d146..78e16bb7e49b9 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md @@ -1,6 +1,8 @@ # Workstream `plugin-boundary`: Plugin Boundary and Web/Wasm Integration -Status: Working plan +Status: In progress. Generic static plugin EP registration has landed and WebGPU uses it; ORT Web, the process-global +lifetime contract, the plugin API gap inventory and the browser bridge are outstanding. Per-package status is under +[Work packages](#work-packages). [WebGPU EP extraction overview](../webgpu_ep_extraction.md) @@ -141,15 +143,41 @@ The bridge should not expose unrelated ORT private implementation details. ## Work packages -1. **Size and latency baselines:** measure `onnxruntime-web` WebAssembly size and inference latency, and native - inference latency, on the non-plugin path while it still exists, since the size and latency completion criteria - compare the plugin path against those numbers. -2. **Static registration core:** implement and contract-test generic static factory registration. -3. **Global lifetime contract:** inventory process-global state and implement safe ownership and teardown rules. -4. **Emscripten prototype:** compile the plugin path statically and run a small model. -5. **Gap inventory triage:** convert private-dependency findings into public API or provider-owned actions. -6. **Browser bridge:** specify and prototype object and lifetime exchange. -7. **Parity and retirement:** run the existing suite through the plugin path and remove the legacy path. +Status reflects the state after static plugin EP registration landed (PR #32395). + +1. **Size and latency baselines** — *In progress.* Measure `onnxruntime-web` WebAssembly size and inference latency, + and native inference latency, on the non-plugin path while it still exists, since the size and latency completion + criteria compare the plugin path against those numbers. Size has been measured, and latency has been measured for + both the web static plugin and the native shared library, but the latency figures are not reproducible: three + measurements of the same model disagree by roughly 3.7x and the baseline itself drifted between sessions. A + controlled re-measurement is outstanding. See + [Measured size comparison](ort_web_static_plugin_migration.md#measured-size-comparison) and + [Measured performance comparison](ort_web_static_plugin_migration.md#measured-performance-comparison) for the web + numbers, and [Native shared-library plugin + performance](static_plugin_ep_registration_validation.md#native-shared-library-plugin-performance) for native. +2. **Static registration core** — *In progress.* Implement and contract-test generic static factory registration. + The facility is implemented and its test results are identical to the internal-EP build on Windows with a real + GPU. What remains is the generic contract test: WebGPU is the only static plugin, so D2's entry-point prefix + uniqueness is unexercised. See [Static Plugin EP Registration](static_plugin_ep_registration_design.md) and + [its validation](static_plugin_ep_registration_validation.md). +3. **Global lifetime contract** — *In progress.* Inventory process-global state and implement safe ownership and + teardown rules. Protobuf ownership is resolved by build-time selection rather than a new ABI hook, and the + adapter's logger wrapper is released with the factory that owns it. The full classification of every subsystem + listed under [Process-global ownership and teardown](#process-global-ownership-and-teardown) is not done, and the + register/unregister/re-register regression is not yet reachable from the existing test binaries. +4. **Emscripten prototype** — *Done.* Compile the plugin path statically and run a small model. A throwaway + prototype ran the full `js/web` WebGPU operator suite in Edge against a WASM `static_plugin` binary: 2152 tests, + all passing. Productionizing that result is + [ORT Web Static Plugin Migration](ort_web_static_plugin_migration.md), not this package. +5. **Gap inventory triage** — *Not started.* Convert private-dependency findings into public API or provider-owned + actions. Input exists: `onnxruntime/core/providers/webgpu/ep/README.md` records WebGPU cleanup and EP default + configuration as gaps, and depends on the `provider-isolation` workstream for the rest. +6. **Browser bridge** — *Not started.* Specify and prototype object and lifetime exchange. The representation + question is still open; see [Open questions](#open-questions). +7. **Parity and retirement** — *In progress.* Run the existing suite through the plugin path and remove the legacy + path. Native parity is demonstrated: zero status differences across the 5894 provider tests common to the static + plugin and internal-EP builds. ORT Web parity depends on the migration above, and removing the legacy path is + explicitly a non-goal until parity is demonstrated there. Packages 1 through 6 can proceed largely in parallel, and legacy-path removal waits for their convergence. That ordering matters for the baselines in particular: the non-plugin path is the comparison, so once package 7 removes it @@ -171,6 +199,11 @@ the numbers can no longer be captured. ## Completion criteria +None of these are met yet; they describe the end state of the whole workstream, not the static-registration +milestone. The criteria most affected by work so far are the two size and latency ones, which now have measurements +but not reproducible ones, and the first, which holds for the `static_plugin` build but not for the ORT Web build +that ships today. + - Static and dynamic WebGPU builds use the same provider implementation and public API boundary. - Factory release and environment teardown cannot shut down process-global state still owned or used by ORT, another factory, or another plugin. From d8f69c183a3c8c07a6619e4c43bcde9dc0a2fc2b Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Fri, 11 Sep 2026 12:51:01 -0700 Subject: [PATCH 075/107] Narrow the plugin boundary workstream open questions to what is actually unknown Both questions were stated as blank unknowns, but each is partly answered. JavaScript-owned WebGPU objects already have a representation: webgpuRegisterDevice imports a GPUDevice through Emscripten's WebGPU bindings and passes the resulting handle, with an instance from _OrtCreateWebGpuInstance, as decimal strings in EP options, and webgpuRegisterBuffer does the same for GPUBuffer. The prototype confirmed these reach the provider unchanged over the plugin path. Restate the question as whether that contract, which relies on Emscripten internals, ORT-owned Wasm exports and stringified pointers rather than a documented public C API, is what an out-of-tree provider should depend on. Of the two gaps the adapter README records, EP default configuration is confirmed to need a public API addition, while WebGPU cleanup is handled on the plugin path by ReleaseEpFactory and is a work package 3 lifetime concern rather than an API gap. Record both so the remaining triage is scoped to provider-isolation findings. --- ...boundary_and_web_integration_workstream.md | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md index 78e16bb7e49b9..3f99c4487dcd4 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md @@ -226,5 +226,20 @@ that ships today. ## Open questions -- What is the stable representation of JavaScript-owned WebGPU objects at the C API boundary? -- Which private-dependency findings require a public plugin EP API addition rather than a provider-owned replacement? +- Is the current JavaScript/Wasm object boundary the one an out-of-tree provider should depend on? The + representation itself is no longer unknown: `GPUDevice` and `GPUBuffer` already cross as Emscripten native + handles. `webgpuRegisterDevice` in `onnxruntime/wasm/post-webgpu.js` imports a device with + `WebGPU.importJsDevice`, pairs it with an instance from the `_OrtCreateWebGpuInstance` export, and passes both as + decimal strings in the `webgpuInstance` and `webgpuDevice` EP options; `webgpuRegisterBuffer` does the equivalent + for buffers and releases them with `_wgpuBufferRelease`. The prototype confirmed these options reach the provider + unchanged over the plugin path. What is unsettled is that this contract is spread across Emscripten's WebGPU + bindings, ORT-owned Wasm exports and stringified pointers in session options, none of which is a documented public + C API, and that ownership and device-loss rules are not written down anywhere. +- Which remaining private-dependency findings require a public plugin EP API addition rather than a provider-owned + replacement? Of the two gaps recorded in `onnxruntime/core/providers/webgpu/ep/README.md`, one is now classified + and one is closed. EP default configuration still needs a public API addition, sketched there as + `SetEpDefaultConfig(ep_name, key, value)`, because it must be settable before a session exists. WebGPU cleanup is + handled on the plugin path by `ReleaseEpFactory`, which clears the kernel registries and WebGPU contexts, with the + legacy `OrtEnv` cleanup compiled out for adapter builds; whether that holds across unregister and re-register is + untested, and is tracked by work package 3 rather than as an API gap. The rest of the triage waits on + `provider-isolation` findings (work package 5). From 8ba6c3f4bbd30cb0bcb93a2d826c1b59f29fdc3a Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Fri, 11 Sep 2026 14:17:03 -0700 Subject: [PATCH 076/107] Close out the plugin boundary workstream open questions Both open questions are resolved, so replace them with decisions and fold the supporting detail into the sections that own the work. The representation of JavaScript-owned WebGPU objects was never really unknown. A GPUDevice crosses as an emdawnwebgpu handle in the webgpuInstance and webgpuDevice EP options, and a GPUBuffer crosses as the tensor data pointer tagged by DATA_LOCATION_GPU_BUFFER, with buffer ownership reference counted in JavaScript and a NaN count marking ORT-created buffers. The decision is to keep that structure, so record it under Browser/Wasm bridge and narrow work package 6 to documenting the rules and settling threading, device loss, validation and Emscripten resilience. Handle identity needs no design work while everything links into one Emscripten module, since emdawnwebgpu's object table is per module and a WebGPU shared library is rejected outright there. On plugin EP API additions, the static plugin case working through unmodified entry points with no new ABI is positive evidence, not merely absent evidence: it is the configuration with the least room to work around a missing API. The adapters' use of ORT internals does not contradict this, because value types and logic can be carried by the provider itself. A gap needs the provider to observe or mutate state inside the running ORT instance. Correct the EP default configuration gap accordingly. The adapter README predates CreateEnvWithOptions and GetEnvConfigEntries, which already let a host pass EP-scoped entries that the factory reads during GetSupportedDevices, as the WebGPU factory does for allow_virtual_devices. What is genuinely uncovered is a binding that never exposes environment creation: OrtEnv is a lazily created singleton whose construction arguments are ignored once it exists, and the Python module builds it without config entries. That also explains why the factory still reads ORT_WEBGPU_EP_ALLOW_SOFTWARE_ADAPTER from the OS environment. --- ...boundary_and_web_integration_workstream.md | 78 ++++++++++++------- 1 file changed, 51 insertions(+), 27 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md index 3f99c4487dcd4..1a502ca5a4726 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md @@ -121,9 +121,22 @@ Likely investigation areas include: - Setting EP default configuration before a session exists, equivalent to the existing `SetCurrentGpuDeviceId`. The adapter's own `Missing parts` section in `onnxruntime/core/providers/webgpu/ep/README.md` is authoritative input -to this inventory rather than speculation. It records two gaps: WebGPU cleanup, which the process-global ownership -and teardown work covers, and EP default configuration, which is missing for both static and shared library builds -and sketches an `OrtApi` addition for it. +to this inventory rather than speculation. It records two gaps, and both have moved since it was written. + +WebGPU cleanup is handled on the plugin path by `ReleaseEpFactory`, which clears the cached kernel registries and +WebGPU contexts, with the legacy `OrtEnv` cleanup compiled out for adapter builds. Whether that survives unregister +and re-register is untested, so it is a lifetime question for the process-global work rather than an API gap. + +EP default configuration is narrower than the README states, because the README predates +`OrtApi::CreateEnvWithOptions` and `OrtEpApi::GetEnvConfigEntries`. A host that creates the environment can already +pass EP-scoped entries using the documented `ep..` prefix, and the factory can read them during +`GetSupportedDevices`; the WebGPU factory already does this for `allow_virtual_devices`. What remains uncovered is a +binding that never exposes environment creation. `OrtEnv` is a lazily created process singleton whose construction +arguments are ignored once it exists, and the Python module builds it without config entries, so a Python caller has +no way to supply them. That is what the `SetEpDefaultConfig(ep_name, key, value)` sketch is really for, and it is +also why the WebGPU factory still reads `ORT_WEBGPU_EP_ALLOW_SOFTWARE_ADAPTER` from the OS environment: an OS +variable is the only channel that works no matter who created the singleton. Whether this is best resolved by a new +API or by letting bindings configure the environment before first use is open; see work package 5. ## Browser/Wasm bridge @@ -141,6 +154,22 @@ The provider repository should own WebGPU-specific behavior. The boundary must d The bridge should not expose unrelated ORT private implementation details. +A mechanism for the first two already exists and ships, and the decision is to keep it rather than design a +replacement. A `GPUDevice` is imported by `webgpuRegisterDevice` in `onnxruntime/wasm/post-webgpu.js` and reaches +the provider as an emdawnwebgpu handle in the `webgpuInstance` and `webgpuDevice` EP options. A `GPUBuffer` is +imported by `webgpuRegisterBuffer` and travels as the tensor's data pointer, tagged by the +`DATA_LOCATION_GPU_BUFFER` argument to `OrtCreateTensor`, which selects a `"WebGPU_Buf"` `OrtMemoryInfo`. Buffer +ownership is reference counted in JavaScript, with a `NaN` count marking an ORT-created buffer that is released +through `Tensor.dispose()` rather than by the bridge. The ORT Web prototype confirmed these reach the provider +unchanged over the plugin path. The remaining work is to document these rules and to settle what the current +mechanism does not address: threading and async behavior, device-loss propagation, validation and error reporting, +and resilience to Emscripten changes. + +Handle identity is not a concern for a statically linked build. Under Emscripten the provider links +`emdawnwebgpu_c`/`emdawnwebgpu_cpp`, whose handles index an object table owned by the Emscripten module's JavaScript +glue, so one module means one table by construction. A WebGPU EP shared library is rejected outright on Emscripten +by `cmake/onnxruntime_providers_webgpu.cmake`, so a second table cannot arise there. + ## Work packages Status reflects the state after static plugin EP registration landed (PR #32395). @@ -170,10 +199,15 @@ Status reflects the state after static plugin EP registration landed (PR #32395) all passing. Productionizing that result is [ORT Web Static Plugin Migration](ort_web_static_plugin_migration.md), not this package. 5. **Gap inventory triage** — *Not started.* Convert private-dependency findings into public API or provider-owned - actions. Input exists: `onnxruntime/core/providers/webgpu/ep/README.md` records WebGPU cleanup and EP default - configuration as gaps, and depends on the `provider-isolation` workstream for the rest. -6. **Browser bridge** — *Not started.* Specify and prototype object and lifetime exchange. The representation - question is still open; see [Open questions](#open-questions). + actions. The working expectation is that few or no additions are needed; see + [Resolved decisions](#resolved-decisions). The one open candidate is reaching EP default configuration from a + language binding that does not expose environment creation, which may be resolved by letting such bindings + configure the environment before first use rather than by a new API. Depends on the `provider-isolation` + workstream for the remaining findings. +6. **Browser bridge** — *Not started.* Specify and prototype object and lifetime exchange. The representation is + settled and kept as is, so the scope is documenting the existing ownership rules and addressing threading and + async behavior, device-loss propagation, validation and error reporting, and resilience to Emscripten changes. + See [Browser/Wasm bridge](#browserwasm-bridge). 7. **Parity and retirement** — *In progress.* Run the existing suite through the plugin path and remove the legacy path. Native parity is demonstrated: zero status differences across the 5894 provider tests common to the static plugin and internal-EP builds. ORT Web parity depends on the migration above, and removing the legacy path is @@ -223,23 +257,13 @@ that ships today. - Should static factories be registered before environment creation, or through environment construction options? Neither. ORT core registers them during `OrtEnv` creation, after the environment is constructed and published. See [Static Plugin EP Registration](static_plugin_ep_registration_design.md). - -## Open questions - -- Is the current JavaScript/Wasm object boundary the one an out-of-tree provider should depend on? The - representation itself is no longer unknown: `GPUDevice` and `GPUBuffer` already cross as Emscripten native - handles. `webgpuRegisterDevice` in `onnxruntime/wasm/post-webgpu.js` imports a device with - `WebGPU.importJsDevice`, pairs it with an instance from the `_OrtCreateWebGpuInstance` export, and passes both as - decimal strings in the `webgpuInstance` and `webgpuDevice` EP options; `webgpuRegisterBuffer` does the equivalent - for buffers and releases them with `_wgpuBufferRelease`. The prototype confirmed these options reach the provider - unchanged over the plugin path. What is unsettled is that this contract is spread across Emscripten's WebGPU - bindings, ORT-owned Wasm exports and stringified pointers in session options, none of which is a documented public - C API, and that ownership and device-loss rules are not written down anywhere. -- Which remaining private-dependency findings require a public plugin EP API addition rather than a provider-owned - replacement? Of the two gaps recorded in `onnxruntime/core/providers/webgpu/ep/README.md`, one is now classified - and one is closed. EP default configuration still needs a public API addition, sketched there as - `SetEpDefaultConfig(ep_name, key, value)`, because it must be settable before a session exists. WebGPU cleanup is - handled on the plugin path by `ReleaseEpFactory`, which clears the kernel registries and WebGPU contexts, with the - legacy `OrtEnv` cleanup compiled out for adapter builds; whether that holds across unregister and re-register is - untested, and is tracked by work package 3 rather than as an API gap. The rest of the triage waits on - `provider-isolation` findings (work package 5). +- Should JavaScript-owned WebGPU objects get a new representation at the C API boundary? No. The existing structure + is kept; see [Browser/Wasm bridge](#browserwasm-bridge) for what it is and what remains to be documented. +- Do statically linked plugin EPs need additions to the plugin EP API? No additions were required. Static linking is + the configuration with the least room to work around a missing API, and it reaches the plugin path through the + unmodified `CreateEpFactories` and `ReleaseEpFactory` entry points, with no new ABI; D3 considered an optional + teardown hook and rejected it. Existing uses of ORT internals by the EP API adapters are not evidence of a gap, + since they are value types and logic that a provider can carry itself. A gap exists only where the provider must + observe or mutate state inside the running ORT instance, which duplication cannot supply. EP default configuration + from a language binding is the one such candidate identified so far; see + [Plugin API gap closure](#plugin-api-gap-closure). Confirming this across the full inventory is work package 5. From 0bd98b7e4e597b0e0815626d6b2da31d51adaea5 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Fri, 11 Sep 2026 16:57:27 -0700 Subject: [PATCH 077/107] docs: consolidate static plugin EP design Make the static registration design self-contained, retain roadmap-level workstream updates, and remove the superseded validation and migration companion documents. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .../ort_web_static_plugin_migration.md | 371 ------------------ ...boundary_and_web_integration_workstream.md | 120 ++---- .../static_plugin_ep_registration_design.md | 103 ++--- ...tatic_plugin_ep_registration_validation.md | 138 ------- 4 files changed, 79 insertions(+), 653 deletions(-) delete mode 100644 docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/ort_web_static_plugin_migration.md delete mode 100644 docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_validation.md diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/ort_web_static_plugin_migration.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/ort_web_static_plugin_migration.md deleted file mode 100644 index 69f533d104247..0000000000000 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/ort_web_static_plugin_migration.md +++ /dev/null @@ -1,371 +0,0 @@ -# ORT Web Static Plugin Migration - -Follow-up design for moving ORT Web onto the statically linked plugin EP path defined in -[Static Plugin EP Registration](static_plugin_ep_registration_design.md). Build and link evidence for the Emscripten -configuration, the remaining migration steps, and the size and performance analysis that constrains them. - -## Emscripten and ORT Web - -Making the statically linked WebGPU EP reachable through the plugin path is what motivates this design, so the -Emscripten configuration was built before landing the change: `--build_wasm --enable_wasm_simd ---enable_wasm_threads --use_webgpu static_plugin --use_webnn --target onnxruntime_webassembly`. It configures, -compiles and links cleanly, producing `ort-wasm-simd-threaded.asyncify.{mjs,wasm}`. - -`static_plugin` is in fact the only plugin kind Emscripten supports — `cmake/onnxruntime_providers_webgpu.cmake` -raises a `FATAL_ERROR` for `shared_lib` there, since there is no runtime library to load. - -The WASM WebGPU jobs that produce the published artifacts pass a bare `--use_webgpu`, which defaults to -`static_lib` (`tools/ci_build/build_args.py` uses `const="static_lib"`), so they still build the internal EP. -`.github/workflows/web.yml` additionally runs a **build-only** `--use_webgpu static_plugin` leg in the -`wasm_Release` job. It keeps the Emscripten registration path compiling, but deliberately publishes nothing: its -output file names are identical to those of the published WebGPU build, and the ORT Web test harness selects the -wasm binary by file name, so the two variants cannot ship side by side. Running the ORT Web suites against the -plugin build therefore waits on switching ORT Web over to it, which is separate work with prerequisites that this -change does not address. See [Migration plan](#migration-plan). - -A green build is *not* sufficient evidence that the registration is live. `CreateStaticPluginEpLibraries()` in -`onnxruntime/core/session/plugin_ep/ep_static_plugins.cc` is guarded by -`#if defined(USE_WEBGPU) && defined(ORT_WEBGPU_STATIC_PLUGIN)`; if that define does not reach ORT core the function -still compiles, simply returning an empty vector, and the build is just as green while registering nothing. The -check that actually discriminates is at the symbol level — ORT core must hold an *undefined* reference to the -prefixed entry points, resolved by the provider library: - -``` -llvm-nm libonnxruntime_session.a | grep WebGpu_ # U WebGpu_CreateEpFactories / U WebGpu_ReleaseEpFactory -llvm-nm libonnxruntime_providers_webgpu.a | grep WebGpu_ # T WebGpu_CreateEpFactories / T WebGpu_ReleaseEpFactory -``` - -Both were observed on the Emscripten build, which confirms the guard was live, the D3 entry-point prefixing worked, -and the link resolved core's reference against the provider. That check validates build, link and registration -wiring; runtime behaviour was established separately by the ORT Web prototype described under -[Migration plan](#migration-plan). - -A practical build note: the WASM build needs `node` on `PATH` — `node_helper.cmake` does a hard `find_program` and -fails configure without it. emsdk bundles a suitable one under `cmake/external/emsdk/node//bin`. - - -## Migration plan - -The intended end state is that the WebGPU EP is always a plugin EP and `--use_webgpu static_lib` goes away -entirely, ORT Web included. Building the EP into the WASM binary as a static plugin, which this change enables, is -only the first half of that. The second half is *selection*, and two gaps stand between the current state and a -working ORT Web on the plugin path. Neither is visible to the linker, so the symbol-level evidence above does not -speak to either. - -Both gaps have since been closed by a throwaway prototype, which ran the full `js/web` WebGPU operator suite in -Edge against a WASM `static_plugin` binary: 2152 tests, all passing. The design below is therefore validated end -to end rather than merely plausible; what remains is productionization, not discovery. The prototype's findings are -recorded inline in the steps that follow. - -**ORT Web selects the EP by name, and that name resolves only to the internal EP.** -`js/web/lib/wasm/session-options.ts` maps the `'webgpu'` backend to `epName = 'WebGPU'`, which reaches -`OrtAppendExecutionProvider` in `onnxruntime/wasm/api.cc` and from there -`OrtApis::SessionOptionsAppendExecutionProvider`. That API is a closed table of hardcoded EP names in -`onnxruntime/core/session/provider_registration.cc`, and its `EpID::WebGPU` case is compiled out under -`ORT_USE_EP_API_ADAPTERS`. On a `static_plugin` build the EP therefore registers at `OrtEnv` creation and is then -unselectable from JS: the call returns "WebGPU execution provider is not supported in this build". This is by -design rather than an oversight — the name-based API is the internal-EP mechanism, and plugin EPs are selected -through `RegisterExecutionProviderLibrary` plus `OrtEpDevice`-based selection — but it does mean ORT Web needs new -plumbing rather than a recompile. - -**No WebGPU `OrtEpDevice` exists under Emscripten.** Emscripten was neither `WIN32`, `LINUX` nor `APPLE`, so -`cmake/onnxruntime_common.cmake` fell through to `core/platform/device_discovery_default.cc`, which discovers only -a CPU device. `Factory::GetSupportedDevicesImpl` in `onnxruntime/core/providers/webgpu/ep/factory.cc` creates an -`OrtEpDevice` only for devices of type `OrtHardwareDeviceType_GPU`. The one fallback, the virtual GPU device in the -same function, is opt-in through the `allow_virtual_devices` environment configuration entry and is deliberately -registered without allocator info, because it exists to back a device-free compile-only session. So even after -selection plumbing is added, `GetEpDevices` would offer nothing for JS to select. - -A workable order: - -1. Make WebGPU visible as a device under Emscripten by adding an Emscripten device discovery implementation. - `cmake/onnxruntime_common.cmake` gains an `Emscripten` arm selecting a new - `core/platform/emscripten/device_discovery.cc`, which reports the CPU device via `GetCpuDeviceFromCPUIDInfo()` - plus a GPU `OrtHardwareDevice` when a synchronous `EM_ASM_INT` check finds `navigator.gpu`. No change to - `Factory::GetSupportedDevicesImpl` is needed: its existing loop filters on `OrtHardwareDeviceType_GPU`, so a - discovered GPU flows through and picks up both allocator infos unmodified. - - The prototype first tried the other option — having the factory synthesize the device itself through - `CreateHardwareDevice` — and it worked, but it is the wrong layer. `GetSupportedDevices` documents its `devices` - argument as "the `OrtHardwareDevice` instances that are available", so that array is the source of truth, and - `CreateHardwareDevice` is documented for devices with no hardware behind them ("e.g., virtual"). A browser GPU - is real hardware that ORT's discovery layer merely failed to enumerate, so the producer is the place to fix it. - Leaving it in the factory would also oblige every future Emscripten plugin EP to repeat the same workaround. - - A capability check is a legitimate form of discovery here. `core/platform/apple/device_discovery.cc` sets the - precedent: it enumerates nothing, it asserts that one GPU exists on Apple Silicon and hardcodes the vendor id. - `navigator.gpu` is a stronger signal than that — an actual runtime capability probe, and the same one ORT Web's - `initEp` uses to decide whether WebGPU is usable. `EM_ASM` from ORT core is likewise established practice - (`core/graph/model.cc`, `core/framework/external_data_loader.cc`), not just an EP-layer habit. - - The honest limitation is that no adapter information is available. `_OrtInit` creates the `OrtEnv` and therefore - triggers discovery, and it runs before `navigator.gpu.requestAdapter()` in ORT Web's `initEp`, so `vendor_id` - and `device_id` are left at 0. That is acceptable: Apple reports no device id either, and browsers mask - `GPUAdapterInfo.vendor` for fingerprinting resistance regardless. The device is deliberately *not* tagged with - `kOrtHardwareDevice_MetadataKey_IsVirtual`, which is what keeps it distinct from the virtual device — that path - is allocator-less by design and cannot back a real session. - - A later change on `main` added a second fallback to `GetSupportedDevicesImpl` alongside the virtual device: when - no GPU device is discovered, `allow_software_adapter` (opt-in via the `kAllowSoftwareAdapterEnvironmentVariable` - environment variable) advertises WebGPU against the *CPU* `OrtHardwareDevice`. It does not change the analysis - above. Both upstream fallbacks work around a device the discovery layer failed to enumerate; they are opt-in - precisely because misreporting a device is not safe to do by default. Under Emscripten the device is genuinely - there and cheaply detectable, so fixing discovery keeps the default path correct and needs no opt-in. The - primary matching loop that our discovered GPU device flows through is unchanged by that work. - - One accepted side effect: `ep_library_internal.cc` creates an internal `WebGpuEpFactory` in a - `USE_WEBGPU && !ORT_USE_EP_API_ADAPTERS` build, so today's `static_lib` ORT Web build starts producing an - `OrtEpDevice` where it previously produced none. That build selects by name and is slated for removal, so the - change is benign. - -2. Expose plugin EP selection in the WASM C API. `g_env` is already available in `api.cc`, so `GetEpDevices` and - `SessionOptionsAppendExecutionProvider_V2` are both reachable. A narrow helper that appends by EP name and - resolves the device internally avoids marshalling arrays of device pointers across the JS boundary. The new - entry point needs to be exported from `cmake/onnxruntime_webassembly.cmake`. - - *Prototype result:* the name-resolving helper is the right shape. Note that exporting it takes three edits, not - one: `EMSCRIPTEN_KEEPALIVE` in `api.h`, the `JSPI_EXPORTS` list in `cmake/onnxruntime_webassembly.cmake` for - JSPI builds, and the `wrapAsyncAPIs` list in `onnxruntime/wasm/pre-async.js` for Asyncify builds. Omitting the - last one is the easy mistake, and Asyncify is the default variant. Reporting matters too: on no match the helper - should raise a status naming the EPs that *are* registered, which is what makes a misconfiguration diagnosable - from JS through `checkLastError`. - -3. Switch `js/web` to the new call. The `BUILD_DEFS.DISABLE_WEBGPU` branch that falls back to the JS EP is - unaffected. Confirm that the EP options ORT Web passes today — `deviceId`, `webgpuInstance`, `webgpuDevice`, - `preferredLayout` and the buffer cache modes — still reach the provider, since `Factory::CreateEpImpl` obtains - them from the session config options. - - *Prototype result:* the options do reach the provider unchanged, and the reason is worth recording. Both paths - derive their config prefix from `OrtSessionOptions::GetProviderOptionPrefix` keyed on the EP name, so V2 lands - the options under exactly the same `ep.webgpuexecutionprovider.*` keys the name-based path used. One JS change - is required though: `session-options.ts` passes the short name `'WebGPU'`, whereas `OrtEpDevice` selection - matches the canonical `'WebGpuExecutionProvider'` returned by `EpDevice_EpName`. - - The productionization question this raises is *conditionality*: is it safe to switch to V2 unconditionally - whenever WebGPU is enabled? It is, provided the WASM build flips at the same time. `BUILD_DEFS.DISABLE_WEBGPU` - is defined in `js/web/script/build.ts` as exactly `!USE_WEBGPU_EP`, so the bundle is already specialized per - WASM flavour by `--webgpu-ep` — the JSEP bundle takes the other branch entirely and is unaffected. Flipping the - WebGPU WASM builds to `static_plugin` (step 4) therefore makes `!DISABLE_WEBGPU` imply `static_plugin` by - construction, and no additional build define or runtime capability probe is needed. What is *not* safe is - landing the JS change without the build change, since the two must move together. - -4. Flip the ORT Web WASM builds to `static_plugin` and extend CI to *test* the result. The two builds are - `Build (simd + threads + WebGPU experimental)` and its JSPI sibling in - `.github/workflows/linux-wasm-ci-build-and-test-workflow.yml`, both of which pass a bare `--use_webgpu` and so - get `static_lib` from `build_args.py`'s `const="static_lib"`. Neither uses `--minimal_build`, so the - minimal-build `FATAL_ERROR` guard does not stand in the way of this migration. A build-only `static_plugin` - leg already runs (see [Emscripten and ORT Web](#emscripten-and-ort-web)), so this path is covered at compile - time; flipping these two builds is what gets it *executed*, because the published artifacts then become the - plugin build and the existing `web_Debug` / `web_Release` suites cover it with no parallel test lane. - - *Prototype note:* the `js/web` test runner cannot currently exercise this configuration end to end, because - `script/test-runner-cli.ts` spawns `script/build` with only `--bundle-mode` and does not forward `--webgpu-ep`. - Setting `npm_config_webgpu_ep` in the environment works as a stopgap, since `script/build.ts` reads it, but a CI - leg should forward the flag properly. Note also that without `--webgpu-ep` the `webgpu` backend silently routes - through JSEP instead, so a leg that omits it would pass while testing nothing relevant. - -## Interaction with the ORT Web reduced-size build options - -The ORT Web WASM builds are size-sensitive and pass a set of size-reduction flags that the local prototype builds -did not. `linux-wasm-ci-build-and-test-workflow.yml` applies `--disable_ml_ops --disable_generation_ops ---disable_types string float4 float8 optional sparsetensor --include_ops_by_config -onnxruntime/wasm/reduced_types.config --enable_reduced_operator_type_support`, and `web.yml` adds `--disable_rtti` -for the release job. Since the migration changes how the WebGPU EP is compiled and linked, each of these was -checked against the plugin path. - -**None of them need adapting.** The findings: - -- *Compile definitions reach the WebGPU target in every mode.* `onnxruntime_providers_webgpu` is created through - `onnxruntime_add_static_library` / `onnxruntime_add_shared_library_module`, both of which call - `onnxruntime_configure_target` → `onnxruntime_set_compile_flags`. That function is where every `DISABLE_*` and - `REDUCED_OPS_BUILD` definition is applied, so the `static_lib`, `static_plugin` and shared-library builds all - receive an identical set. There is no propagation gap introduced by the plugin boundary. - -- *Operator and type reduction never applied to WebGPU, before or after.* - `op_registration_utils.get_kernel_registration_files()` hardcodes the CPU registration files (plus CUDA when - requested); WebGPU is not in the list, so `--include_ops_by_config` does not rewrite WebGPU kernel - registrations. `onnxruntime/wasm/reduced_types.config` is `!no_ops_specified_means_all_ops_are_required` — it - performs global *type* reduction only, with no op exclusion — and that mechanism works through the - `op_kernel_type_control` macros, which neither `core/providers/webgpu` nor `contrib_ops/webgpu` uses. Its saving - is entirely CPU-EP-side and is unchanged by the plugin switch. This is a pre-existing gap, not a regression. - -- *The type and ML/generation op flags are inert for WebGPU.* No source under `core/providers/webgpu` or - `contrib_ops/webgpu` references `DISABLE_ML_OPS`, `DISABLE_GENERATION_OPS`, `DISABLE_SPARSE_TENSORS`, - `DISABLE_OPTIONAL_TYPE`, `DISABLE_FLOAT8_TYPES`, `DISABLE_FLOAT4_TYPES` or `DISABLE_STRING_TYPE`. - -- *`--disable_rtti` is safe on the plugin path.* Neither `include/onnxruntime/ep` (the EP API adapters) nor - `onnxruntime/core/session/plugin_ep` uses `dynamic_cast` or `typeid`. - -One consequence is worth recording for the *shared library* plugin EP, which is out of scope here but shares this -code. `onnxruntime_c_api.h` contains no `DISABLE_*` guards, so the `OrtApi` struct layout is invariant under these -flags and the ABI is stable. `onnxruntime_cxx_api.h` does guard some `Ort::Value` members (the sparse tensor -methods) — an EP DLL built with a different `--disable_types` than its host would therefore see a different C++ -header surface, but since those wrappers are inline over stable function pointers this is a source-compatibility -concern rather than an ABI one. For `static_plugin` everything is compiled in one tree with one set of defines, so -the question does not arise. - -**What is *not* covered by any existing flag** is the plugin machinery itself. Relative to `static_lib`, the -`static_plugin` build adds `core/providers/webgpu/ep/*` (excluded from the source list in the `static_lib` branch -of `onnxruntime_providers_webgpu.cmake`), inlines the header-only EP API adapters into every WebGPU translation -unit, and reaches the `OrtGraph` / `OrtNode` C API graph views through `OrtEp::GetCapability`. Note that -`core/session/plugin_ep` is *not* part of that delta: `onnxruntime_session.cmake` only excludes it for -`onnxruntime_MINIMAL_BUILD`, so today's WASM builds already compile it in. - -That also means the only existing lever that trims plugin-EP machinery is `--minimal_build`, which -`static_plugin` currently rejects outright. If measurement shows the size delta matters, lifting that restriction -(already listed as a follow-up) is the mechanism to reach parity — not a new size-reduction flag. - -### Measured size comparison - -The ORT Web npm package ships **two** WebGPU WASM artifacts, not one, and both were measured. The release path -(`npm-packaging-pipeline.yml` → `templates/web-ci.yml` → `templates/linux-wasm-ci.yml`) and the GitHub Actions path -(`web.yml` → `linux-wasm-ci-build-and-test-workflow.yml`) pass an identical flag set for the WebGPU legs, which -differ from each other only in their exception/stack-switching mode: - -| Leg | Artifact | Distinguishing flag | -| --- | --- | --- | -| `wasm_inferencing_webgpu` | `ort-wasm-simd-threaded.asyncify.{wasm,mjs}` | `--enable_wasm_api_exception_catching` | -| `wasm_inferencing_webgpu_jspi` | `ort-wasm-simd-threaded.jspi.{wasm,mjs}` | `--enable_wasm_jspi` | - -Four clean from-scratch builds were run (both EP modes × both legs) with the exact release flag set: `Release`, -SIMD + threads, WebNN on, `--disable_rtti`, `--target onnxruntime_webassembly`, and the full reduced-size set -above. Compressed sizes are included because that is what is actually delivered over the wire; they were produced -with gzip level 9 and brotli quality 11. - -`ort-wasm-simd-threaded.asyncify.wasm`: - -| Metric | `static_lib` | `static_plugin` | Delta | -| --- | ---: | ---: | ---: | -| raw | 26,549,423 B | 26,487,397 B | **−62,026 B (−0.234%)** | -| gzip | 6,537,701 B | 6,566,088 B | **+28,387 B (+0.434%)** | -| brotli | 3,953,625 B | 3,971,550 B | **+17,925 B (+0.453%)** | - -`ort-wasm-simd-threaded.jspi.wasm`: - -| Metric | `static_lib` | `static_plugin` | Delta | -| --- | ---: | ---: | ---: | -| raw | 16,627,577 B | 16,687,113 B | **+59,536 B (+0.358%)** | -| gzip | 4,099,914 B | 4,126,147 B | **+26,233 B (+0.640%)** | -| brotli | 2,651,643 B | 2,674,520 B | **+22,877 B (+0.863%)** | - -The accompanying `.mjs` files are byte-identical between the two EP modes in both legs (53,300 B asyncify, 51,204 B -JSPI); their compressed sizes differ by at most 23 B. - -**Every delta is under 1%, which is the substantive conclusion.** Two details are worth recording so the numbers -are not over-read: - -- *The raw sign is not stable across the two artifacts.* `static_plugin` is marginally smaller raw on asyncify but - marginally larger raw on JSPI. An earlier revision of this document reported only the asyncify raw number and - generalised it to "no size regression"; that generalisation was too broad. The asyncify measurement itself - reproduces well — it was −61,968 B when first taken and −62,026 B in this from-scratch rerun at a later commit. - -- *Compressed size grows slightly in both legs*, by +0.4% to +0.9%, so the direction is consistent once the - artifacts are compressed. Why compressed size can rise while raw size falls was not investigated; no - symbol-level or per-section attribution was done, and none of the earlier speculation about *which* code the - plugin build drops or adds has been measured. Given the magnitude, this was not pursued further. - -Both configurations compiled cleanly with every reduced-size flag in both legs, which empirically confirms the -compatibility analysis above. For reference, the same `static_plugin` asyncify configuration without any -reduced-size flags is 40.9 MB, so those flags are worth ~15 MB and remain essential regardless of EP mode. - -### `--enable_wasm_api_exception_catching` and the EP API adapters - -The one option that *did* interact with the plugin path is `--enable_wasm_api_exception_catching`. It is not a -size-reduction flag in the same sense as the others, but it is part of the same CI flag set and it exposed a real -bug. - -`onnxruntime_webassembly.cmake` compiles only `wasm/api.cc` and `core/session/onnxruntime_c_api.cc` with -`-sDISABLE_EXCEPTION_CATCHING=0`; every other translation unit keeps Emscripten's default, where `catch` clauses -are compiled so that they never match. Exceptions still propagate — they are simply only catchable at the C API -boundary. - -The EP API adapters in `include/onnxruntime/ep/adapter/` are header-only and are therefore inlined into WebGPU -translation units that have catching disabled. `OpKernelInfo::GetAttr()` / `GetAttrs()` were implemented as a -`try` around the throwing `Ort::` C++ wrappers, converting `Ort::Exception` into a `Status`. That makes a *missing -optional attribute* — ordinary control flow for `GetAttrOrDefault()` — depend on catching an exception. With -catching disabled the `catch` never ran, so the exception escaped `GetAttrOrDefault()` entirely and surfaced from -session creation as e.g. `ERROR_CODE: 6, ERROR_MESSAGE: No attribute with name:'extrapolation_value'is defined.` - -This reproduced as 949 `suite0` failures on the reduced-size `static_plugin` build while the same build passed on -the CPU backend, the reduced-size `static_lib` build passed, and the full-op `static_plugin` build passed — the -failure required plugin + WebGPU + API-only exception catching together. - -The fix is to not use exceptions for control flow in the adapters: `GetAttr()` / `GetAttrs()` now call the -non-throwing `OrtApi::KernelInfoGetAttribute*` functions directly and convert the returned `OrtStatus*` into a -`Status`. **General rule: header-only EP API adapter code must not rely on catching exceptions**, because a plugin -EP may be compiled with exception catching disabled. The remaining `catch` blocks in `include/onnxruntime/ep` -(`common.h`, `get_capability_utils.h`, `adapter/kernel_registry.h`) sit on genuinely exceptional callback -boundaries rather than on control-flow paths; when catching is disabled those exceptions still reach the outer C -API boundary and are reported, only with a less specific message. - -With that fix the reduced-size `static_plugin` build passes `suite0` on the WebGPU backend in full: 2152 tests, -all passing, matching the `static_lib` control run on the identical flag set. - -### Measured performance comparison - -The same two binaries were compared on inference performance in Edge, using the test runner's `--perf` mode -(`test-runner-cli model -b=webgpu -e=edge -P=`) against two synthetic models chosen to isolate the two -places the plugin path could plausibly cost something: - -- **dispatch-bound** — 300 chained tiny elementwise ops (`Mul`/`Add`/`Sub`) on a `[1, 1024]` tensor. Almost no GPU - work per node, so the run time is dominated by per-node host-side work, which is exactly what now goes through - the C API adapters. -- **compute-bound** — 16 chained `[512, 512]` `MatMul`s. Host-side per-node cost is negligible against the GPU - work. - -Three runs per model per build. **The two builds must be sampled interleaved** — one round is -`plugin/dispatch, plugin/compute, lib/dispatch, lib/compute`, repeated — rather than all samples of one build -followed by all samples of the other. This machine drifts: a build that has just finished, a warm browser, or a -warm GPU shifts timings by more than the effect being measured, and sequential sampling aliases that drift onto -the build under test. The table reports the median of the three per-run P50 values. - -| Metric | `static_lib` | `static_plugin` | Delta | -| --- | ---: | ---: | ---: | -| dispatch-bound, per-run P50 | 11.20 ms | 12.00 ms | +0.8 ms (+7%) | -| compute-bound, per-run P50 | 7.70 ms | 7.80 ms | +0.1 ms (within noise) | - -Reading of these numbers: - -- **Execution is not measurably slower once GPU work dominates.** The compute-bound samples overlap - (`static_plugin` 7.80 / 7.80 / 7.40 against `static_lib` 7.90 / 7.00 / 7.70 — the plugin build's best sample - beats two of the three control samples), so the difference is not resolvable at this sample count. -- **A dispatch-bound graph pays roughly 7%.** That is the cost of routing per-node kernel work through the C API - adapters instead of a direct in-process virtual call. Unlike the compute-bound case this one does separate - cleanly — the plugin build's slowest control sample is still faster than its own fastest sample. It only shows - up when nodes do essentially no GPU work, which is the worst case by construction rather than a representative - one. Amortized over ~300 nodes it is on the order of 3 µs per node — but see the caveat below before relying on - that figure. -- **Session creation costs about 3%** (measured pre-merge: 1136 → 1172 ms and 1250 → 1290 ms for the two models). - This covers plugin EP registration, `GetCapability` and kernel creation through the C API. The absolute figure - is dominated by fetching and parsing the model, so the true relative cost of the plugin machinery within session - creation is higher than 3% — but it is tens of milliseconds once, not per inference. - -Two cautions on methodology: - -- The *total* wall time the test runner prints for `suite0` is not a usable metric. It varied between 3 min 15 s - and 4 min 04 s for the **same** `static_lib` binary across two runs. Only the second figure it prints — net test - time, excluding the per-model `before all` session-creation hooks — is stable enough to compare, and by that - measure the two builds are equal. An early single-sample comparison of the total figure suggested a 29% - regression that does not exist. -- Timings taken with `--webgpu.profiling.mode=default` are inflated by the timestamp queries and must never be - compared against timings taken without it. - -**The per-node magnitude is not reliably established, and a controlled re-measurement is outstanding.** Three -separate measurements of the same 300-node dispatch-bound model disagree by roughly 3.7x: +0.80 ms (2.7 µs/node) -above, +1.50 ms (5.0 µs/node) in a later four-model sweep that fitted 4.76 µs/node across 150/300/600 nodes, and -+3.00 ms (10.0 µs/node) in a four-side build comparison. The `static_lib` baseline itself moved from 7.90 ms to -11.90 ms for the same model on the same machine within a day, so build-to-build and session drift dominates the -spread, and Edge's 0.1 ms `performance.now()` floor without cross-origin isolation limits resolution further. The -three runs also did not use the same statistic (per-run P50 versus median-of-round-minimum). A re-measurement -should record the commit of each arm, the full build flag set, the model definitions, the statistic, the -interleaved sampling order and the GPU submit counts, so the number can be reproduced. - -What *is* robust is the sign and the mechanism. The plugin side was slower in 26 of 26 paired rounds in the sweep -and 6 of 6 in a three-way run, and GPU submit counts were byte-identical on every side, model and round, so the -cost is CPU-side per-kernel work rather than extra queue work. The same three-way run also excludes the -JavaScript side: a `mismatch` arm (plugin bundle against a `static_lib` wasm) came out level with `static_lib`, so -the `_OrtAppendExecutionProviderV2` call and the EP-name lookup are free and the cost is inside the -`static_plugin` wasm's per-kernel C++ path. **Where exactly is not yet attributed** — an `--enable_wasm_profiling` -build would give named frames, and the native root cause recorded under [Native shared-library plugin -performance](static_plugin_ep_registration_validation.md#native-shared-library-plugin-performance) is the leading -candidate. diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md index 1a502ca5a4726..cd90a252aaa2f 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/plugin_boundary_and_web_integration_workstream.md @@ -1,8 +1,7 @@ # Workstream `plugin-boundary`: Plugin Boundary and Web/Wasm Integration -Status: In progress. Generic static plugin EP registration has landed and WebGPU uses it; ORT Web, the process-global -lifetime contract, the plugin API gap inventory and the browser bridge are outstanding. Per-package status is under -[Work packages](#work-packages). +Status: In progress. Generic static plugin EP registration is implemented in PR #32395, and WebGPU can be built +through it with `--use_webgpu static_plugin`. ORT Web migration and the remaining lifetime work are outstanding. [WebGPU EP extraction overview](../webgpu_ep_extraction.md) @@ -33,8 +32,8 @@ does not own WebGPU kernels, Dawn, shader tooling, or the provider's standalone ## Generic static plugin registration -Add an ORT facility that accepts statically linked plugin factory entry points. It should reuse the existing dynamic -plugin path after library loading and symbol lookup. +The static plugin facility accepts linked factory entry points and reuses the existing dynamic plugin path after +library loading and symbol lookup. The design must address: @@ -49,9 +48,8 @@ The design must address: The facility must be generic. Validation with a non-WebGPU test plugin is deferred: WebGPU is currently the only statically linked plugin EP, so this is tracked as a test gap rather than a completed criterion. -The detailed design is in [Static Plugin EP Registration](static_plugin_ep_registration_design.md), with supporting -evidence in [Static Plugin EP Registration: Validation](static_plugin_ep_registration_validation.md) and the ORT Web -follow-up in [ORT Web Static Plugin Migration](ort_web_static_plugin_migration.md). +The detailed design and validation summary are in +[Static Plugin EP Registration](static_plugin_ep_registration_design.md). ## Process-global ownership and teardown @@ -121,22 +119,9 @@ Likely investigation areas include: - Setting EP default configuration before a session exists, equivalent to the existing `SetCurrentGpuDeviceId`. The adapter's own `Missing parts` section in `onnxruntime/core/providers/webgpu/ep/README.md` is authoritative input -to this inventory rather than speculation. It records two gaps, and both have moved since it was written. - -WebGPU cleanup is handled on the plugin path by `ReleaseEpFactory`, which clears the cached kernel registries and -WebGPU contexts, with the legacy `OrtEnv` cleanup compiled out for adapter builds. Whether that survives unregister -and re-register is untested, so it is a lifetime question for the process-global work rather than an API gap. - -EP default configuration is narrower than the README states, because the README predates -`OrtApi::CreateEnvWithOptions` and `OrtEpApi::GetEnvConfigEntries`. A host that creates the environment can already -pass EP-scoped entries using the documented `ep..` prefix, and the factory can read them during -`GetSupportedDevices`; the WebGPU factory already does this for `allow_virtual_devices`. What remains uncovered is a -binding that never exposes environment creation. `OrtEnv` is a lazily created process singleton whose construction -arguments are ignored once it exists, and the Python module builds it without config entries, so a Python caller has -no way to supply them. That is what the `SetEpDefaultConfig(ep_name, key, value)` sketch is really for, and it is -also why the WebGPU factory still reads `ORT_WEBGPU_EP_ALLOW_SOFTWARE_ADAPTER` from the OS environment: an OS -variable is the only channel that works no matter who created the singleton. Whether this is best resolved by a new -API or by letting bindings configure the environment before first use is open; see work package 5. +to this inventory rather than speculation. It records two gaps: WebGPU cleanup, which the process-global ownership +and teardown work covers, and EP default configuration, which is missing for hosts that do not expose environment +configuration. ## Browser/Wasm bridge @@ -154,64 +139,21 @@ The provider repository should own WebGPU-specific behavior. The boundary must d The bridge should not expose unrelated ORT private implementation details. -A mechanism for the first two already exists and ships, and the decision is to keep it rather than design a -replacement. A `GPUDevice` is imported by `webgpuRegisterDevice` in `onnxruntime/wasm/post-webgpu.js` and reaches -the provider as an emdawnwebgpu handle in the `webgpuInstance` and `webgpuDevice` EP options. A `GPUBuffer` is -imported by `webgpuRegisterBuffer` and travels as the tensor's data pointer, tagged by the -`DATA_LOCATION_GPU_BUFFER` argument to `OrtCreateTensor`, which selects a `"WebGPU_Buf"` `OrtMemoryInfo`. Buffer -ownership is reference counted in JavaScript, with a `NaN` count marking an ORT-created buffer that is released -through `Tensor.dispose()` rather than by the bridge. The ORT Web prototype confirmed these reach the provider -unchanged over the plugin path. The remaining work is to document these rules and to settle what the current -mechanism does not address: threading and async behavior, device-loss propagation, validation and error reporting, -and resilience to Emscripten changes. - -Handle identity is not a concern for a statically linked build. Under Emscripten the provider links -`emdawnwebgpu_c`/`emdawnwebgpu_cpp`, whose handles index an object table owned by the Emscripten module's JavaScript -glue, so one module means one table by construction. A WebGPU EP shared library is rejected outright on Emscripten -by `cmake/onnxruntime_providers_webgpu.cmake`, so a second table cannot arise there. - ## Work packages -Status reflects the state after static plugin EP registration landed (PR #32395). - -1. **Size and latency baselines** — *In progress.* Measure `onnxruntime-web` WebAssembly size and inference latency, - and native inference latency, on the non-plugin path while it still exists, since the size and latency completion - criteria compare the plugin path against those numbers. Size has been measured, and latency has been measured for - both the web static plugin and the native shared library, but the latency figures are not reproducible: three - measurements of the same model disagree by roughly 3.7x and the baseline itself drifted between sessions. A - controlled re-measurement is outstanding. See - [Measured size comparison](ort_web_static_plugin_migration.md#measured-size-comparison) and - [Measured performance comparison](ort_web_static_plugin_migration.md#measured-performance-comparison) for the web - numbers, and [Native shared-library plugin - performance](static_plugin_ep_registration_validation.md#native-shared-library-plugin-performance) for native. -2. **Static registration core** — *In progress.* Implement and contract-test generic static factory registration. - The facility is implemented and its test results are identical to the internal-EP build on Windows with a real - GPU. What remains is the generic contract test: WebGPU is the only static plugin, so D2's entry-point prefix - uniqueness is unexercised. See [Static Plugin EP Registration](static_plugin_ep_registration_design.md) and - [its validation](static_plugin_ep_registration_validation.md). -3. **Global lifetime contract** — *In progress.* Inventory process-global state and implement safe ownership and - teardown rules. Protobuf ownership is resolved by build-time selection rather than a new ABI hook, and the - adapter's logger wrapper is released with the factory that owns it. The full classification of every subsystem - listed under [Process-global ownership and teardown](#process-global-ownership-and-teardown) is not done, and the - register/unregister/re-register regression is not yet reachable from the existing test binaries. -4. **Emscripten prototype** — *Done.* Compile the plugin path statically and run a small model. A throwaway - prototype ran the full `js/web` WebGPU operator suite in Edge against a WASM `static_plugin` binary: 2152 tests, - all passing. Productionizing that result is - [ORT Web Static Plugin Migration](ort_web_static_plugin_migration.md), not this package. -5. **Gap inventory triage** — *Not started.* Convert private-dependency findings into public API or provider-owned - actions. The working expectation is that few or no additions are needed; see - [Resolved decisions](#resolved-decisions). The one open candidate is reaching EP default configuration from a - language binding that does not expose environment creation, which may be resolved by letting such bindings - configure the environment before first use rather than by a new API. Depends on the `provider-isolation` - workstream for the remaining findings. -6. **Browser bridge** — *Not started.* Specify and prototype object and lifetime exchange. The representation is - settled and kept as is, so the scope is documenting the existing ownership rules and addressing threading and - async behavior, device-loss propagation, validation and error reporting, and resilience to Emscripten changes. - See [Browser/Wasm bridge](#browserwasm-bridge). -7. **Parity and retirement** — *In progress.* Run the existing suite through the plugin path and remove the legacy - path. Native parity is demonstrated: zero status differences across the 5894 provider tests common to the static - plugin and internal-EP builds. ORT Web parity depends on the migration above, and removing the legacy path is - explicitly a non-goal until parity is demonstrated there. +1. **Size and latency baselines — In progress:** measure `onnxruntime-web` WebAssembly size and inference latency, + and native inference latency, on the non-plugin path while it still exists. +2. **Static registration core — Implemented in PR #32395:** contract-test generic static factory registration. + Validation with more than one static plugin remains outstanding. +3. **Global lifetime contract — In progress:** inventory process-global state and implement safe ownership and + teardown rules. +4. **Emscripten prototype — Done:** compile the plugin path statically and run it in a browser. Production ORT Web + migration remains separate work. +5. **Gap inventory triage — Not started:** convert private-dependency findings into public API or provider-owned + actions. +6. **Browser bridge — Not started:** specify and prototype object and lifetime exchange. +7. **Parity and retirement — In progress:** run the existing suite through the plugin path and remove the legacy + path after ORT Web parity is demonstrated. Packages 1 through 6 can proceed largely in parallel, and legacy-path removal waits for their convergence. That ordering matters for the baselines in particular: the non-plugin path is the comparison, so once package 7 removes it @@ -233,11 +175,6 @@ the numbers can no longer be captured. ## Completion criteria -None of these are met yet; they describe the end state of the whole workstream, not the static-registration -milestone. The criteria most affected by work so far are the two size and latency ones, which now have measurements -but not reproducible ones, and the first, which holds for the `static_plugin` build but not for the ORT Web build -that ships today. - - Static and dynamic WebGPU builds use the same provider implementation and public API boundary. - Factory release and environment teardown cannot shut down process-global state still owned or used by ORT, another factory, or another plugin. @@ -257,13 +194,6 @@ that ships today. - Should static factories be registered before environment creation, or through environment construction options? Neither. ORT core registers them during `OrtEnv` creation, after the environment is constructed and published. See [Static Plugin EP Registration](static_plugin_ep_registration_design.md). -- Should JavaScript-owned WebGPU objects get a new representation at the C API boundary? No. The existing structure - is kept; see [Browser/Wasm bridge](#browserwasm-bridge) for what it is and what remains to be documented. -- Do statically linked plugin EPs need additions to the plugin EP API? No additions were required. Static linking is - the configuration with the least room to work around a missing API, and it reaches the plugin path through the - unmodified `CreateEpFactories` and `ReleaseEpFactory` entry points, with no new ABI; D3 considered an optional - teardown hook and rejected it. Existing uses of ORT internals by the EP API adapters are not evidence of a gap, - since they are value types and logic that a provider can carry itself. A gap exists only where the provider must - observe or mutate state inside the running ORT instance, which duplication cannot supply. EP default configuration - from a language binding is the one such candidate identified so far; see - [Plugin API gap closure](#plugin-api-gap-closure). Confirming this across the full inventory is work package 5. + +## Open questions +- Which private-dependency findings require a public plugin EP API addition rather than a provider-owned replacement? diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md index bfb6528c57059..8b38a161d05d8 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -1,8 +1,7 @@ # Static Plugin EP Registration -Status: Implemented in PR #32395. ORT Web's migration onto this path remains follow-up work; see -[ORT Web Static Plugin Migration](ort_web_static_plugin_migration.md). Detailed design supporting -[Plugin Boundary and Web/Wasm Integration](plugin_boundary_and_web_integration_workstream.md). +Status: Implemented in draft PR #32395. ORT Web's migration onto this path remains follow-up work. Detailed design +supporting [Plugin Boundary and Web/Wasm Integration](plugin_boundary_and_web_integration_workstream.md). ## Purpose @@ -10,8 +9,8 @@ Define how a plugin execution provider (EP) that is linked into the host binary that statically linked and dynamically loaded plugin EPs share one provider implementation and one public API boundary. -WebGPU is the first consumer. The immediate deliverable is to move the statically linked WebGPU build off -`EpLibraryInternal` and onto the plugin EP path. +WebGPU is the first consumer. This change adds a statically linked WebGPU configuration that uses the plugin EP path, +alongside the existing static build that uses `EpLibraryInternal`. ## Decision summary @@ -70,6 +69,31 @@ boundary. - Support for static plugin EPs in minimal builds. `cmake/onnxruntime_session.cmake` excludes all of `core/session/plugin_ep/` when `onnxruntime_MINIMAL_BUILD` is set. +## Static plugin contract + +- A statically linked provider exposes the standard `CreateEpFactories` and `ReleaseEpFactory` entry points under a + provider-specific prefix that is unique within the host binary. Their signatures and factory ownership rules are + unchanged from the shared-plugin ABI. +- ORT core discovers those entry points through its build-guarded registry. Applications do not register linked-in + providers explicitly. +- Registration occurs during process-singleton `OrtEnv` creation, after the environment is published but before + other threads can obtain it. Direct callers of `onnxruntime::Environment::Create` do not receive static plugin + registration. +- `GetSupportedDevices` may call environment-resolving `OrtApi` functions on the creating thread. It runs before the + current provider's devices are committed, so `GetEpDevices` does not include those devices and may include only + static providers registered earlier in the registry. +- ORT releases factories through `ReleaseEpFactory` during environment teardown. A statically linked provider must + not release process-global state owned by the host. +- Minimal builds reject static plugin EP configuration until the required plugin infrastructure is available there. + +### Failure behavior + +`EpLibraryStaticPlugin` releases any factories created before `CreateEpFactories` reports a failure. Duplicate +registration names are rejected by the existing `Environment::RegisterExecutionProviderLibrary` path, and other +factory or device-registration errors retain that path's diagnostics. If automatic static registration fails, +`OrtEnv::GetOrCreateInstance` unpublishes and destroys the partially created singleton before returning the error, so +another thread cannot observe an environment whose static registrations are incomplete. + ## Decisions ### D1: Registration mechanism is a link-time registry in ORT core @@ -89,9 +113,9 @@ Alternatives considered: site in every statically linked host and introduces a window in which `GetEpDevices` reports no devices. Revisit when an out-of-tree provider needs static linking. -### D2: Entry point names are macro-parameterized +### D2: Entry point names depend on the linkage mode -`onnxruntime/core/providers/webgpu/ep/api.cc` is parameterized by `ORT_PLUGIN_EP_ENTRY_POINT_PREFIX`. +`onnxruntime/core/providers/webgpu/ep/api.cc` uses `ORT_PLUGIN_EP_STATICALLY_LINKED` to select the entry point names. The shared build must continue to export the unprefixed `CreateEpFactories` and `ReleaseEpFactory`, because `EpLibraryPlugin::Load` resolves those exact names. The static build emits prefixed variants, for example @@ -107,9 +131,6 @@ state it initialized and must tear it down, whereas a statically linked plugin s owns its lifetime and shuts it down itself. `ORT_PLUGIN_EP_STATICALLY_LINKED` is defined by the static plugin configuration only; see D7 for where it is defined and what else it controls. -Rationale: a new optional entry point cannot be relied upon, because an older ORT would load a newer plugin and -silently ignore it. Behavior that varies with the host version is worse than a compile-time decision. - The legacy static WebGPU cleanup block in `onnxruntime/core/session/ort_env.cc` is already guarded by `defined(USE_WEBGPU) && !defined(ORT_USE_EP_API_ADAPTERS)`. The static plugin configuration defines `ORT_USE_EP_API_ADAPTERS`, so the block compiles out on its own and needs no edit. It must be kept for the default @@ -167,8 +188,6 @@ The `dynamic_plugin_ep_infra` naming becomes inaccurate. Renaming is deferred to ### D6: Static plugin EPs are registered after the environment is published -This is the subtlest decision and the one that constrains the others. - **Problem.** The natural insertion point is next to `CreateAndRegisterInternalEps` inside `Environment::Initialize`. Registration enumerates devices, which calls provider code: @@ -198,8 +217,8 @@ therefore runs against a fully constructed and published environment. Properties: -- The entire `OrtApi` behaves normally for provider code. `TryGetInstance` finds a published instance, and a - re-entrant `CreateEnv` acquires the recursive lock and increments the reference count. +- Environment-resolving `OrtApi` calls can safely re-enter on the creating thread. `TryGetInstance` finds a published + instance, and a re-entrant `CreateEnv` acquires the recursive lock and increments the reference count. - No new contract for provider authors, and nothing for a future API author to remember. The constraints that do apply to `GetSupportedDevices` are the ones dynamic plugin EP registration already imposes, now documented on `OrtEpFactory::GetSupportedDevices` in `onnxruntime_ep_c_api.h`. @@ -298,41 +317,27 @@ internal WebGPU EP is compiled out and the parameter reaches `ORT_UNUSED_PARAMET This does not affect the `.virtual` registration-name suffix or the `allow_virtual_devices` environment configuration entry, which remain in use for dynamically registered plugin libraries. -## Risks and open questions - -- Whether adapter-compiled WebGPU objects link into ORT core without duplicate-symbol or ODR problems. One real - instance was found and resolved: the `ORT_API_MANUAL_INIT` `detect_mismatch` guard, see D7. Otherwise resolved: - `onnxruntime_test_all` and `onnxruntime_provider_test` both link cleanly against the statically linked EP. -- Provider process-global state surviving unregistration, since static linking has no unload event. Covered by the - re-registration regression case in D3. Not yet exercised: `OrtEnv` is a reference-counted process singleton and the - unit test main holds a reference for the lifetime of the process, so a second `Environment` construction (and - therefore a second `CreateEpFactories` call) is not reachable from the existing test binaries. This needs either a - dedicated single-test process or a test that fully releases and recreates the environment. -- Whether any shipped ORT Web configuration uses an extended-minimal build, which excludes plugin EP infrastructure - entirely. -- No `std::condition_variable` is paired with `OrtEnv::m_`, so making it recursive does not require - `condition_variable_any`. This is an invariant to preserve rather than an open question: `m_` is a private static - member of `OrtEnv`, so every use is confined to `onnxruntime/core/session/ort_env.cc`, and all of them are plain - `std::lock_guard` acquisitions. A future `std::condition_variable` waiting on `m_` would not compile against a - recursive mutex, so the constraint is enforced by the type system rather than by review. -- The existing rejection of `onnxruntime_BUILD_DAWN_SHARED_LIBRARY` together with the adapters. Initially left - rejecting both plugin kinds. -- Binary size and dead-code elimination for statically linked providers, per the workstream completion criteria. -- Validation with a non-WebGPU static plugin EP is deferred, not completed. `CreateStaticPluginEpLibraries()` - currently has exactly one entry, so the design is generic by construction but unexercised with more than one - provider. The specific untested contract is entry-point prefix uniqueness within a host binary (D2): with a - single provider a prefix collision cannot occur. This is a test gap, not a known defect. +## Limitations and follow-ups -## Validation summary - -The static plugin configuration builds and runs on Windows with a real GPU, and its test results are identical to -the internal-EP build: across the 5894 `onnxruntime_provider_test` cases common to both, zero status differences. -Linux GCC and Emscripten are covered by build-only CI legs. The known gaps are operator coverage on the plugin path -for internal-only white-box tests, environment re-creation, and any runtime execution of the Emscripten build. +| Item | State | +| --- | --- | +| Environment recreation | Untested. Existing unit-test binaries retain the process-singleton `OrtEnv`; a dedicated process is needed to exercise register, run, tear down, recreate, and run again. | +| Multiple static plugin EPs | Untested. WebGPU is the only registry entry, so prefix uniqueness and registration ordering are not covered with two providers. | +| Minimal builds | Unsupported and rejected at configure time because they exclude plugin EP infrastructure. | +| ORT Web production migration | Deferred. The shipping Wasm configuration still uses the internal EP path. | +| Dawn shared-library mode | Remains incompatible with the EP API adapter configuration. | +| Binary size and dead-code elimination | Deferred to ORT Web migration, where the shipped Wasm artifact can be measured. | -Full results, CI job inventory, symbol-level checks and native performance measurements are in -[Static Plugin EP Registration: Validation](static_plugin_ep_registration_validation.md). +## Validation summary -ORT Web's migration onto this path, including the Emscripten build status, the remaining migration steps and the -size and performance analysis, is in -[ORT Web Static Plugin Migration](ort_web_static_plugin_migration.md). +| Configuration | Result | Evidence and remaining gap | +| --- | --- | --- | +| Windows static plugin, real GPU | Build and tests pass | Zero status differences across the 5894 `onnxruntime_provider_test` cases common to the static-plugin and internal-EP builds. | +| Windows shared plugin | Build and tests pass | The DLL continues to export the unprefixed entry points required by `EpLibraryPlugin`. | +| Linux static plugin, GCC | Build passes | Build-only CI covers GCC and Python wheel linkage; GPU execution is not exercised. | +| Emscripten static plugin | Build passes | Build-only CI and symbol inspection verify linked entry points. A local prototype passed 2152 browser WebGPU tests, but the shipping artifacts and blocking CI still use the internal EP path. | +| Minimal build | Rejected at configure time | Prevents a provider from being linked into a binary that omits its registration infrastructure. | + +The adapter configuration excludes 12 operator cases that depend on internal WebGPU test infrastructure, in addition +to white-box and disabled tests. Re-registration after complete `OrtEnv` teardown and registration of more than one +static plugin EP also remain untested. diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_validation.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_validation.md deleted file mode 100644 index c87fba6eb6e19..0000000000000 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_validation.md +++ /dev/null @@ -1,138 +0,0 @@ -# Static Plugin EP Registration: Validation - -Evidence for [Static Plugin EP Registration](static_plugin_ep_registration_design.md). That document defines the -design and its contracts; this one records what was built, what was run, and what is still untested. - -## Test results - -Verified on Windows with `--use_webgpu static_plugin`, Debug: - -- `onnxruntime_test_all` and `onnxruntime_provider_test` build and link. -- The statically linked WebGPU plugin EP is registered by ORT core during `Ort::Env` creation, and its `OrtEpDevice` - is enumerated and selectable, confirming the D6 publish-then-register ordering resolves the self-deadlock. -- `InferenceSessionTests.WebGpuVirtualDeviceCompileOnlyEndToEnd` and - `InferenceSessionTests.WebGpuVirtualDeviceRejectedWithoutCompileOnly` pass, which covers session creation and EP - instantiation end to end through the public V2 API on the static plugin path. - -Operator execution was initially not verifiable locally: the original development machine was a virtual machine with -no real GPU, so only the virtual WebGPU device was surfaced and non-compile-only sessions were rejected up front. This -was confirmed to be a property of the host rather than of static linking by building the shared library plugin -configuration (`--use_webgpu shared_lib`) on the same host and running the same tests: it passed and failed exactly -the same set (`WebGpuVirtualDeviceCompileOnlyEndToEnd` and `WebGpuVirtualDeviceRejectedWithoutCompileOnly` pass; -`WebGpuCompileOnlySkipsFinalization` and `TestStrictShapeInference` fail with the identical virtual-device rejection -message). - -This gap has since been closed on a machine with a real GPU (NVIDIA RTX 5060 Ti). The `static_plugin` and internal-EP -configurations were built from the same tree, RelWithDebInfo, differing only in the EP path, and their test results -compared: - -- `onnxruntime_provider_test`: 0 failures in both. **Across the 5894 tests common to both builds there were zero - status differences.** -- `onnxruntime_test_all`: 1965 passed in both, no differences. All three `InferenceSessionTests.WebGpu*` pass on the - static plugin build, including `WebGpuCompileOnlySkipsFinalization`, confirming its earlier failure was indeed a - property of the GPU-less host. - -## Known test coverage difference on the plugin path - -`cmake/onnxruntime_unittests.cmake` gates the `test/providers/webgpu/*` sources on -`onnxruntime_USE_WEBGPU AND NOT onnxruntime_USE_EP_API_ADAPTERS`, so those tests are **absent** from any adapter -build, including `static_plugin`. They are white-box tests that include internal WebGPU EP headers and therefore -cannot compile against the adapter boundary by construction. - -Of the 44 tests this excludes, 16 are `DISABLED_` and never run, and most of the rest assert on EP internals -(`WebGpuContextTest`, `ActivationCacheKeyTest`). However, 12 of them — `HardSwish_WebGPU` and `MatMul2BitsWebGpu` — -are genuine operator tests, and they currently run **only** on the internal EP path. This is a known coverage gap of -the plugin path. Closing it would mean rewriting them as EP-agnostic operator tests so both paths execute them; that -is deliberately left out of this change to keep it focused. - -Separately, the `InferenceSessionTests.WebGpu*` virtual-device tests are deliberately *paired* rather than shared -between the two builds, since each drives a different factory. `WebGpuVirtualDeviceCompileOnlyEndToEnd` and -`WebGpuVirtualDeviceRejectedWithoutCompileOnly` are the plugin-build counterparts of `WebGpuEpFactoryVirtualDevice` -and `WebGpuEpFactoryRejectsVirtualDeviceWithoutCompileOnly`. Only `WebGpuCompileOnlyUsesNoOpAllocator` has no -counterpart, because it asserts an internal allocator *type* through `dynamic_cast`, which is inherently -unobservable across the ABI boundary. - -The shared library plugin configuration was also re-verified after the D7 change, since `include/onnxruntime/ep/api.h` -is shared with the CUDA plugin EP. It builds, and `onnxruntime_providers_webgpu.dll` still exports exactly the -unprefixed `CreateEpFactories` and `ReleaseEpFactory` required by `EpLibraryPlugin::Load`, per D2. - -## CI coverage - -No pre-existing CI leg built this configuration, since both new CMake options default to `OFF`. Three legs were -added: - -- `webgpu_static_plugin_build_x64_RelWithDebInfo` in `.github/workflows/windows_webgpu.yml`, which builds and runs the - tests on a GPU-equipped pool. This closes the operator execution gap above. Note that it must not set - `onnxruntime_BUILD_DAWN_SHARED_LIBRARY=ON`, which is incompatible with `onnxruntime_USE_EP_API_ADAPTERS`. -- `build-linux-webgpu-static-plugin-x64-release` in `.github/workflows/linux_webgpu.yml`, build only, mirroring its - sibling job. This is the only GCC coverage of the EP adapters, and the only coverage of the `onnxruntime_python.cmake` - change, hence `--build_wheel` is kept. -- A build-only `--use_webgpu static_plugin` step in the `wasm_Release` job of `.github/workflows/web.yml`, which - keeps the Emscripten registration path compiling. It publishes no artifacts; see - [ORT Web Static Plugin Migration](ort_web_static_plugin_migration.md#emscripten-and-ort-web) for why, and for what - that leg does and does not prove. - -The first run of the Linux job surfaced `-Werror=maybe-uninitialized` errors. The EP adapter `OpKernelInfo` defines its -attribute accessors inline in the header, so GCC can see their failure and exception paths, whereas the in-tree -`OpKernelInfo` hides them behind an out-of-line definition. Any value that is only read when the attribute lookup -succeeded must be value-initialized. - -More generally, this job is the first time the EP adapters have ever been compiled with GCC, and it has surfaced -several successive batches of `-Werror` diagnostics — including false positives inside `absl::InlinedVector` and, in -optimized builds, `-Werror=array-bounds` on `std::string` concatenation chains that the adapter headers cause to be -inlined more aggressively. Because ninja stops at the first failure and reports only the targets already in flight, -**each CI run reveals just one batch**. When touching this configuration, enumerate the full set locally in a single -pass — an AlmaLinux 8 container with `gcc-toolset-14` reproduces the CI compiler exactly — rather than discovering -them one CI round trip at a time. Fixes that are correct on their own merits are applied at the source; irreducible -false positives use a suppression scoped to the `onnxruntime_providers_webgpu` target, GCC only, and conditional on -`onnxruntime_USE_EP_API_ADAPTERS`, so non-adapter builds stay strict. This mirrors what the CUDA plugin EP does in -`cmake/onnxruntime_providers_cuda_plugin.cmake`. - -That full local enumeration has since been done, and it is worth recording what it found: of the ~340 warnings GCC 14 -emits across the whole `static_plugin` build, **exactly one is attributable to ORT's own source** — the -`-Warray-bounds` false positive in `contrib_ops/webgpu/quantization/matmul_nbits_mlp.cc`, fixed by building the -expression with `MakeString` instead of a `std::string` concat chain. Everything else lives in Dawn, ONNX, -flatbuffers, or the telemetry dependency, in targets that do not use `-Werror`. The practical lesson is that the -adapters do not make ORT's code broadly warning-prone; they perturb inlining at a small number of specific sites, -so a targeted source fix is almost always available and preferable to a blanket target-wide suppression. - -When validating such a fix locally, always rebuild the *unfixed* source through the same harness first and confirm it -still errors. A harness that silently fails to exercise the warning is indistinguishable from a successful fix. - - -## Native shared-library plugin performance - -The latency criterion covers two shapes of plugin, and the ORT Web numbers under [Measured performance -comparison](ort_web_static_plugin_migration.md#measured-performance-comparison) settle only the static one. A -native `--use_webgpu shared_lib` A/B against a built-in baseline measured the plugin slower by ~8% on a -dispatch-bound model and 4-6% on Qwen3.5-0.8B, fitting roughly **67 µs per `Run` plus 1.2 µs per node**. The -reproducibility caveat recorded for the web numbers applies here too; these figures should be re-taken with the -commit, flags and harness recorded. - -One control is what licenses reading the delta as boundary cost at all: matching build flags do not prove two -executables perform alike, so both binaries were also run on the CPU EP, where no plugin is loaded. That gave a -ratio of minima of 1.001 — but only single-threaded. Run with default threading the same control showed the -plugin executable 24% *faster*, which would have been actively misleading. - -**Root cause: redundant per-kernel work, not boundary crossing.** 1.2 µs/node is ~4,400 cycles, far more than a -cross-DLL indirect call can account for, so both arms were CPU-profiled. `ep::adapter::CreateTensorFromApiValue` -rebuilds an `onnxruntime::Tensor` — nine C API round trips, an allocator-name `std::string`, a `TensorShape` -allocation — for every input and output of every node on every `Run`, because the adapter's `OpKernelContext` is -constructed fresh per `Compute()` and caches nothing beyond it. The adapter symbols are absent from the built-in -profile entirely, `Tensor`/`TensorShape` work doubles, and the extra CPU (+0.53 ms/rep) accounts for the entire -extra wall latency (+0.42 ms/rep). - -That makes the cost look **reducible rather than intrinsic**, which matters for how the latency criterion is -settled: negotiating a tolerance around a fixable defect would be the wrong order. The obvious candidates are -caching the `Tensor` wrapper across `Compute()` calls, and avoiding the allocator-name `std::string` and the -`TensorShape` allocation on the per-node path. This is also the leading candidate for the unattributed web -per-node cost, since both boundaries charge per kernel node. - -Two gaps remain in the native numbers. Session-creation cost was not compared — the plugin arm must also discover -and load the DLL, so it is likely worse in relative terms. And both arms ran with default WebGPU options; graph -capture in particular targets exactly the per-dispatch cost identified above and is worth a follow-up. - -**The plugin path silently loses the WebGPU EP's `Api` profiling events.** The built-in arm emits 15,300 `Api` -category events under `-p` and the plugin arm emits none. This is an observability regression independent of -latency, and it also means ORT's own profiler cannot perform this A/B: under `-p` the built-in arm measures -*slower* and the result inverts. From 5fd229e35799e5547d223826b736517a46dc9d43 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Fri, 11 Sep 2026 17:33:41 -0700 Subject: [PATCH 078/107] docs: streamline static plugin EP design Restructure the document around scope, contract, and decisions; remove repeated implementation history; and preserve the key rationale, limitations, and validation results in a shorter form. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .../static_plugin_ep_registration_design.md | 298 ++++++------------ 1 file changed, 89 insertions(+), 209 deletions(-) diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md index 8b38a161d05d8..d5c4d9ebb7405 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -1,16 +1,19 @@ # Static Plugin EP Registration -Status: Implemented in draft PR #32395. ORT Web's migration onto this path remains follow-up work. Detailed design -supporting [Plugin Boundary and Web/Wasm Integration](plugin_boundary_and_web_integration_workstream.md). +Status: Implemented in PR #32395. ORT Web's migration onto this path remains follow-up work. This design +supports [Plugin Boundary and Web/Wasm Integration](plugin_boundary_and_web_integration_workstream.md). -## Purpose +## Context and scope -Define how a plugin execution provider (EP) that is linked into the host binary is registered with ONNX Runtime, so -that statically linked and dynamically loaded plugin EPs share one provider implementation and one public API -boundary. +A provider linked into the host could not previously use the plugin EP path. `EpLibraryPlugin` discovers factory +entry points in a shared library, while `EpLibraryInternal` wraps an in-tree `IExecutionProvider`. Neither accepts +linked factory entry points. -WebGPU is the first consumer. This change adds a statically linked WebGPU configuration that uses the plugin EP path, -alongside the existing static build that uses `EpLibraryInternal`. +This design adds that missing discovery path. Static and dynamic plugins then share the same factory lifecycle and +public API boundary. WebGPU is the first consumer because Emscripten cannot use its shared-module build. + +This design does not add a public static-registration API, support minimal builds, remove the EP API adapters, or +remove WebGPU's existing `EpLibraryInternal` path. Those changes remain follow-up work. ## Decision summary @@ -35,40 +38,6 @@ flowchart LR Factories --> Devices[Register factories and devices in Environment] ``` -## Background - -Before this change, the two `EpLibrary` implementations relevant here were: - -- `EpLibraryPlugin` (`onnxruntime/core/session/plugin_ep/ep_library_plugin.cc`) loads a shared library, resolves the - `CreateEpFactories` and `ReleaseEpFactory` symbols, and drives the factory lifecycle. -- `EpLibraryInternal` (`onnxruntime/core/session/plugin_ep/ep_library_internal.cc`) wraps an in-tree - `IExecutionProvider`, including `EpLibraryInternal::CreateWebGpuEp`. - -Neither accepted factory entry points directly, so a provider compiled into the host could not reach the plugin -path. `EpLibraryStaticPlugin` (`onnxruntime/core/session/plugin_ep/ep_library_static_plugin.h`) is what this design -adds to close that gap. - -This blocked ORT Web specifically. `cmake/onnxruntime_providers_webgpu.cmake` raises a `FATAL_ERROR` for the WebGPU -shared-module build under Emscripten, so static linking is the only way a Wasm build can ever reach the plugin -boundary. - -## Goals - -- Register a plugin EP that is linked into the host, reusing the existing plugin registration path after the point - where symbol lookup would occur. -- Present statically linked plugin code with the same runtime conditions as dynamically loaded plugin code. -- Require no change to provider sources between the two linkage modes. -- Avoid new host call sites for in-tree providers. -- Keep the shared-library path's existing ABI and runtime behavior unchanged. - -## Non-goals - -- A public C API for registering static plugin EPs. Deferred until a provider is genuinely out-of-tree. -- Removal of the EP API adapters. Tracked separately. -- Removal of the direct `IExecutionProvider` WebGPU path. That is the final step of the workstream, not this change. -- Support for static plugin EPs in minimal builds. `cmake/onnxruntime_session.cmake` excludes all of - `core/session/plugin_ep/` when `onnxruntime_MINIMAL_BUILD` is set. - ## Static plugin contract - A statically linked provider exposes the standard `CreateEpFactories` and `ReleaseEpFactory` entry points under a @@ -86,105 +55,69 @@ boundary. not release process-global state owned by the host. - Minimal builds reject static plugin EP configuration until the required plugin infrastructure is available there. -### Failure behavior +### Failure handling -`EpLibraryStaticPlugin` releases any factories created before `CreateEpFactories` reports a failure. Duplicate -registration names are rejected by the existing `Environment::RegisterExecutionProviderLibrary` path, and other -factory or device-registration errors retain that path's diagnostics. If automatic static registration fails, -`OrtEnv::GetOrCreateInstance` unpublishes and destroys the partially created singleton before returning the error, so -another thread cannot observe an environment whose static registrations are incomplete. +`EpLibraryStaticPlugin` releases factories created before `CreateEpFactories` reports a failure. Existing +`Environment::RegisterExecutionProviderLibrary` diagnostics cover duplicate names and factory or device errors. If +automatic registration fails, `OrtEnv::GetOrCreateInstance` unpublishes and destroys the partial singleton before +returning the error. Other threads therefore cannot observe incomplete static registration. ## Decisions ### D1: Registration mechanism is a link-time registry in ORT core -ORT core holds a hand-written, `#if`-guarded list of statically linked plugin EP entry points, mirroring the shape of -`EpLibraryInternal::CreateInternalEps`. A new `EpLibraryStaticPlugin` accepts the entry points directly and reuses -`EpLibraryPlugin`'s factory lifecycle logic without the dynamic-library load and unload steps. - -Rationale: every host — `onnxruntime_test_all`, the Python bindings, `onnxruntime.dll`, and Wasm — gets the provider -with no per-host call site. +ORT core holds a hand-written, `#if`-guarded list of linked factory entry points. `EpLibraryStaticPlugin` accepts +those entry points directly and reuses `EpLibraryPlugin` after its library-loading and symbol-resolution steps. +This gives every host the provider without adding a host-specific registration call. Alternatives considered: -- CMake-generated registry using an X-macro. Rejected: added build-system machinery for generality that may never be - needed while providers remain in-tree. -- A public `RegisterStaticExecutionProviderLibrary` C API called by each host. Rejected for now: it requires a call - site in every statically linked host and introduces a window in which `GetEpDevices` reports no devices. Revisit - when an out-of-tree provider needs static linking. +- A CMake-generated registry adds machinery that is not justified while providers remain in-tree. +- A public registration API requires every host to call it and creates a period in which `GetEpDevices` has no + static devices. Revisit this option when an out-of-tree provider needs static linking. ### D2: Entry point names depend on the linkage mode -`onnxruntime/core/providers/webgpu/ep/api.cc` uses `ORT_PLUGIN_EP_STATICALLY_LINKED` to select the entry point names. - -The shared build must continue to export the unprefixed `CreateEpFactories` and `ReleaseEpFactory`, because -`EpLibraryPlugin::Load` resolves those exact names. The static build emits prefixed variants, for example -`WebGpu_CreateEpFactories`, so that multiple static plugins can coexist in one binary. +Shared plugins continue to export `CreateEpFactories` and `ReleaseEpFactory`, the names resolved by +`EpLibraryPlugin::Load`. Static plugins use provider-prefixed names such as `WebGpu_CreateEpFactories`, allowing +multiple providers to coexist in one binary. `ORT_PLUGIN_EP_STATICALLY_LINKED` selects the names in provider code. ### D3: No new EP library ABI hook for teardown -All teardown remains in `ReleaseEpFactory`. Process-global shutdown that belongs to the host — currently -`google::protobuf::ShutdownProtobufLibrary()` in `onnxruntime/core/providers/webgpu/ep/api.cc` — is guarded by -`#if !defined(ORT_PLUGIN_EP_STATICALLY_LINKED)`, so it is compiled **in** for the shared-library build and **out** -for the statically linked build. The polarity follows ownership: a shared plugin module owns the process-global -state it initialized and must tear it down, whereas a statically linked plugin shares protobuf with the host, which -owns its lifetime and shuts it down itself. `ORT_PLUGIN_EP_STATICALLY_LINKED` is defined by the static plugin -configuration only; see D7 for where it is defined and what else it controls. +Factory teardown remains in `ReleaseEpFactory`. Host-owned process-global teardown, currently +`google::protobuf::ShutdownProtobufLibrary()`, is compiled out when `ORT_PLUGIN_EP_STATICALLY_LINKED` is defined. A +shared module owns the state it initializes; a static plugin shares that state with the host, which owns its lifetime. -The legacy static WebGPU cleanup block in `onnxruntime/core/session/ort_env.cc` is already guarded by -`defined(USE_WEBGPU) && !defined(ORT_USE_EP_API_ADAPTERS)`. The static plugin configuration defines -`ORT_USE_EP_API_ADAPTERS`, so the block compiles out on its own and needs no edit. It must be kept for the default -internal-EP WebGPU build. +The existing WebGPU cleanup in `ort_env.cc` remains for the internal-EP build. It already compiles out for the static +plugin configuration because that configuration defines `ORT_USE_EP_API_ADAPTERS`. -Consequence to test for: static linking has no library unload, so provider process-global state survives -unregistration. Registering, running, unregistering, re-registering, and running again is the primary regression case. +Static linking has no library unload, so process-global state survives unregistration. A complete regression test +must register, run, tear down the environment, recreate it, and run again. ### D4: One new durable build option -- `onnxruntime_USE_EP_API_ADAPTERS` is unchanged. It is transitional and is retired when WebGPU moves fully onto the - public plugin API. -- `onnxruntime_WEBGPU_STATIC_PLUGIN` is new and durable. It selects linkage into the host rather than a loadable - module. -- `onnxruntime_WEBGPU_LINKED_INTO_HOST` is derived and used at the linkage sites. - -`onnxruntime_USE_EP_API_ADAPTERS` currently conflates three meanings: compiling against the adapters, building a -separate module, and registering by path in tests. The static plugin configuration answers the first yes and the -other two no, so only the linkage and test sites move to the derived option. The global `add_compile_definitions` in -`cmake/CMakeLists.txt` stays as-is, and no provider sources change. - -`cmake/onnxruntime_providers_webgpu.cmake` gains a third arm. The Emscripten and `onnxruntime_BUILD_CACHE` -`FATAL_ERROR`s narrow to the shared-module case, which is what unblocks ORT Web. - -`build.py` gains `--use_webgpu static_plugin`. - -`onnxruntime_WEBGPU_STATIC_PLUGIN` validates its prerequisites at configure time and fails with `FATAL_ERROR` if -they are not met: it requires `onnxruntime_USE_WEBGPU` and `onnxruntime_USE_EP_API_ADAPTERS`, and it rejects a -minimal build. The last one is a temporary guard rather than a permanent restriction. A minimal build excludes -`core/session/plugin_ep` from `onnxruntime_session_srcs`, and `Environment::CreateAndRegisterStaticPluginEps` is -compiled out with it, but `cmake/onnxruntime_providers_webgpu.cmake` still builds and links the provider. The -combination would therefore produce a binary containing the WebGPU EP that never registers it, with nothing -reporting the problem at configure, build or run time. Allowing minimal builds to use a static plugin EP is -worthwhile and needs the registration path to be available there first; the exclusion is currently justified by -provider-bridge dependencies, and only two of the files under `plugin_ep` are provider-bridge, so the subset may be -separable. Until then a loud configure error is preferable to a silently EP-less binary. - -The CUDA plugin is unaffected: it is gated by `onnxruntime_BUILD_CUDA_EP_AS_PLUGIN` and sets -`ORT_USE_EP_API_ADAPTERS` as a private target compile definition, never referencing the CMake option. CUDA's split -between role (`BUILD_CUDA_EP_AS_PLUGIN`) and mechanism (`ORT_USE_EP_API_ADAPTERS`) is the in-tree template for -WebGPU's eventual cleanup. +`onnxruntime_WEBGPU_STATIC_PLUGIN` selects linked-in plugin mode, and `build.py` exposes it as +`--use_webgpu static_plugin`. The derived `onnxruntime_WEBGPU_LINKED_INTO_HOST` controls linkage sites. + +The option requires WebGPU and the EP API adapters. It rejects minimal builds because they exclude the plugin EP +registration code while still linking the provider, which would otherwise create a binary in which WebGPU is present +but never registered. Emscripten and build-cache restrictions continue to apply only to the shared-module mode. + +`onnxruntime_USE_EP_API_ADAPTERS` remains transitional. Provider sources and the CUDA plugin configuration are +unchanged. ### D5: Test infrastructure treats the library path as optional -In `onnxruntime/test/unittest_util/test_dynamic_plugin_ep.cc`, `InitializationConfig::ep_library_path` becomes -optional. When absent, library registration is skipped because ORT core has already performed it, and the RAII -registration handle is left empty. The existing handle deleter already tolerates an empty handle. Device selection, -de-duplication, and factory creation are unchanged. +`InitializationConfig::ep_library_path` becomes optional. When it is absent, tests skip library registration because +ORT core has already registered the static plugin. The existing RAII handle supports this empty state; device +selection, de-duplication, and factory creation are unchanged. Virtual devices are enabled through the `allow_virtual_devices` environment configuration entry (`kOrtEnvAllowVirtualDevices`) supplied at environment creation, rather than the `.virtual` registration-name suffix. The suffix is unavailable because ORT core chooses the registration name for statically linked providers. -The `dynamic_plugin_ep_infra` naming becomes inaccurate. Renaming is deferred to a separate mechanical change. +A shared CMake helper applies the static-plugin test definitions to both `onnxruntime_test_all` and +`onnxruntime_provider_test`, where the operator tests run. ### D6: Static plugin EPs are registered after the environment is published @@ -204,119 +137,53 @@ OrtEnv::GetOrCreateInstance() lock(m_) self-deadlock ``` -`Environment::CreateAndRegisterInternalEps` already documents this hazard, and the internal WebGPU factory avoids it -by capturing `allow_virtual_devices` at construction. A plugin factory cannot use that workaround, because the public -API is all it has. - -The problem is not confined to one function. Plugin EPs receive the full `OrtApi`, so any API that resolves the -environment is affected, including `CreateEnv`, which re-enters `GetOrCreateInstance` and deadlocks on the same mutex. +The internal WebGPU factory avoids this hazard by capturing configuration at construction. A plugin factory cannot: +its only access is through `OrtApi`, and any environment-resolving API can re-enter the same mutex. **Decision.** `OrtEnv::GetOrCreateInstance` publishes `p_instance_` and takes its own reference *before* invoking `Environment::CreateAndRegisterStaticPluginEps`, and `OrtEnv::m_` becomes a `std::recursive_mutex`. Registration therefore runs against a fully constructed and published environment. -Properties: +Guarantees: -- Environment-resolving `OrtApi` calls can safely re-enter on the creating thread. `TryGetInstance` finds a published - instance, and a re-entrant `CreateEnv` acquires the recursive lock and increments the reference count. -- No new contract for provider authors, and nothing for a future API author to remember. The constraints that do - apply to `GetSupportedDevices` are the ones dynamic plugin EP registration already imposes, now documented on - `OrtEpFactory::GetSupportedDevices` in `onnxruntime_ep_c_api.h`. -- No race. Other threads block on `m_` for the duration, which is already true across `Environment::Create`. -- The reference count cannot reach zero mid-construction, because the creating thread's reference is taken first. -- Provider sources need no conditional compilation for the two linkage modes. +- Environment-resolving API calls can re-enter on the creating thread and find the published instance. +- Other threads remain blocked until registration completes. +- The creating thread's reference prevents destruction during registration. +- Provider code and its `GetSupportedDevices` contract remain the same for both linkage modes. Costs and limits: -- Failure during registration requires explicit teardown of the just-published instance. -- A recursive mutex is normally undesirable. Here it states the actual invariant: the environment creation path can - legitimately re-enter the environment accessor on the same thread. Static plugin EP registration is the only reason - `OrtEnv::m_` is recursive; it was a `std::mutex` before this change and no other code path requires recursion. -- A static plugin EP's `GetSupportedDevices` runs against an environment in which its own `OrtEpDevice` instances are - not yet registered, so `OrtApi::GetEpDevices` returns an incomplete list. This is not specific to static linking: - `EpInfo::Create` calls `GetSupportedDevices` before `RegisterExecutionProviderLibrary` appends to - `execution_devices_`, so a dynamically registered plugin EP sees the same thing. Registration also proceeds one - library at a time, so with more than one static plugin EP the Nth would observe the first N-1, making the visible - set depend on `CreateStaticPluginEpLibraries` ordering. Accepted as a side effect. -- Callers of `onnxruntime::Environment::Create` do not get static plugin EPs. In the tree this is one production call - site plus tests and orttraining sample binaries. +- Registration failure must explicitly tear down the published instance. +- `OrtEnv::m_` becomes recursive solely to permit same-thread re-entry during static registration. +- During `GetSupportedDevices`, `GetEpDevices` cannot include the current provider and can include only static + providers registered earlier. Dynamic registration has the same current-provider limitation. +- Direct callers of `onnxruntime::Environment::Create` deliberately bypass singleton registration, often to control + logging or threading in tests, and do not receive static plugin EPs. - Provider code that starts a thread which calls `CreateEnv` and then joins it will deadlock. This is already true of any code running inside `Environment::Create`. -The call to `Environment::CreateAndRegisterStaticPluginEps` therefore lives in `ort_env.cc`. Static plugin -registration is a process-singleton concern tied to the `OrtEnv` lifetime, so the singleton wrapper is its correct -home. +`CreateAndRegisterStaticPluginEps` therefore runs from the `OrtEnv` singleton wrapper, not from +`Environment::Initialize`. `CreateAndRegisterInternalEps` remains in `Environment::Initialize`, so directly created +environments retain internal EP devices. Alternatives considered: -- Thread-local pointer to the environment under construction, consulted by `GetEnvConfigEntries`. Rejected: it - addresses one function, while the reachable surface is the whole `OrtApi`. -- Defer device enumeration until after environment creation. Rejected: lazy initialization moves registration - failures from environment creation to the first `GetEpDevices` call, which is a worse place to report them. -- Register outside the lock after publishing. Rejected: another thread can observe a published environment whose - static EPs are not yet registered, and avoiding that requires a second lock and a completion barrier. - -`onnxruntime::Environment` remains directly constructible, and tests rely on that. `OrtEnv::GetOrCreateInstance` -returns any existing instance and ignores the logging manager, threading options, and configuration entries passed by -later callers, so a test needing specific threading or logging configuration must bypass it. Those tests are -deliberately opting out of process-singleton semantics and should not receive static plugin EPs. +- A thread-local environment pointer fixes only selected APIs, not the full `OrtApi` surface. +- Deferred device enumeration reports registration failures from `GetEpDevices` instead of environment creation. +- Registration outside the lock lets other threads observe incomplete registration unless another barrier is added. ### D7: The statically linked build does not use manual C++ API initialization -`include/onnxruntime/ep/api.h` force-enables `ORT_API_MANUAL_INIT` around its include of `onnxruntime_cxx_api.h`, and -`onnxruntime::ep::ApiInit` calls `Ort::InitApi(ort_api)`. That is required for a plugin EP shared library, which must -not call `OrtGetApiBase()` itself. - -`onnxruntime_cxx_api.h` emits `#pragma detect_mismatch("ORT_API_MANUAL_INIT", ...)` on MSVC, so every translation -unit linked into one binary must agree. The statically linked plugin EP is linked with ORT core and with test code, -neither of which uses manual initialization, so forcing it on produces `LNK2038` for every EP object file. +Shared plugins use `ORT_API_MANUAL_INIT` because they must initialize the C++ API from the host-supplied `OrtApi`. +MSVC requires every translation unit in one binary to agree on that mode, so using it in a statically linked plugin +causes `LNK2038` against ORT core and test code. -The build therefore defines `ORT_PLUGIN_EP_STATICALLY_LINKED` on the statically linked plugin EP target. Under that -macro `ep/api.h` includes `onnxruntime_cxx_api.h` unmodified and `ApiInit` skips `Ort::InitApi`. The C++ API then -default-initializes from `OrtGetApiBase()->GetApi(ORT_API_VERSION)`, which resolves in-process and yields the same -`OrtApi` that `ApiInit` would have installed, because the EP and ORT are the same binary and therefore the same -version. `onnxruntime::ep::ApiPtrs` is still populated from the `OrtApiBase*` that ORT passes to `CreateEpFactories`, -so the EP's own API access is unchanged. +For a static plugin, `ORT_PLUGIN_EP_STATICALLY_LINKED` leaves `onnxruntime_cxx_api.h` in its normal mode and skips +`Ort::InitApi`. `OrtGetApiBase()` resolves in the same binary and returns the same-version API. The EP-specific +`ApiPtrs` are still populated from the `OrtApiBase*` passed to `CreateEpFactories`. The macro is generic rather than WebGPU-specific because `ep/api.h` is shared with the CUDA plugin EP. -## Implementation map - -| Decision | Owning code | -| --- | --- | -| D1 static registry | `core/session/plugin_ep/ep_static_plugins.cc`, `ep_library_static_plugin.h` | -| D1 shared factory lifecycle | `core/session/plugin_ep/ep_library_plugin.cc`, factored out for reuse | -| D2 entry point prefixing | `core/providers/webgpu/ep/api.cc`, `include/onnxruntime/ep/api.h` | -| D3 process-global ownership | `core/providers/webgpu/ep/api.cc` `ReleaseEpFactory` | -| D4 build option | `cmake/CMakeLists.txt`, `cmake/onnxruntime_providers_webgpu.cmake`, `tools/ci_build/build.py` | -| D5 optional library path | `test/unittest_util/test_dynamic_plugin_ep.cc` | -| D6 registration timing | `core/session/ort_env.cc`, `Environment::CreateAndRegisterStaticPluginEps` | -| D7 C++ API initialization | `include/onnxruntime/ep/api.h` | - -Test definitions are applied to `onnxruntime_provider_test` as well as `onnxruntime_test_all`, via a shared -`onnxruntime_set_webgpu_plugin_ep_test_definitions` cmake function. `onnxruntime_provider_test` holds the operator -tests, so without this the plugin path has no operator test coverage. It also needs -`ORT_UNIT_TEST_HAS_WEBGPU_STATIC_PLUGIN_EP` in its own right, because that macro is what enables the -`allow_virtual_devices` environment configuration entry at environment creation in `test_main.cc`. - -## Follow-up cleanup - -`Environment::CreateAndRegisterInternalEps` stays where it is, at the end of `Environment::Initialize`. Internal EP -factories are ORT-core code that does not re-enter the `OrtEnv` singleton, so they are not subject to the constraint -in D6. Moving them would also remove EP devices from every directly constructed `Environment`, and `InferenceSession` -reads `Environment::GetOrtEpDevices` directly. For static plugin EPs the resulting gap is unimplemented new -functionality; for internal EPs it would be a silent regression. - -The `allow_virtual_devices` parameter threaded through `EpLibraryInternal::CreateInternalEps` exists only for the -internal WebGPU EP factory, which cannot query the environment at `GetSupportedDevices` time. The comment in -`CreateAndRegisterInternalEps` documents that constraint. Both can be deleted once WebGPU is no longer an internal EP: -the remaining internal EPs are CPU, kept deliberately as a special case, and DML, which is no longer maintained. -Neither takes the parameter. In a build that defines `ORT_USE_EP_API_ADAPTERS` this is already true, since the -internal WebGPU EP is compiled out and the parameter reaches `ORT_UNUSED_PARAMETER`. - -This does not affect the `.virtual` registration-name suffix or the `allow_virtual_devices` environment -configuration entry, which remain in use for dynamically registered plugin libraries. - ## Limitations and follow-ups | Item | State | @@ -327,17 +194,30 @@ configuration entry, which remain in use for dynamically registered plugin libra | ORT Web production migration | Deferred. The shipping Wasm configuration still uses the internal EP path. | | Dawn shared-library mode | Remains incompatible with the EP API adapter configuration. | | Binary size and dead-code elimination | Deferred to ORT Web migration, where the shipped Wasm artifact can be measured. | +| Internal WebGPU cleanup | Remove `allow_virtual_devices` plumbing from `CreateAndRegisterInternalEps` after retiring the internal WebGPU path. | ## Validation summary | Configuration | Result | Evidence and remaining gap | | --- | --- | --- | -| Windows static plugin, real GPU | Build and tests pass | Zero status differences across the 5894 `onnxruntime_provider_test` cases common to the static-plugin and internal-EP builds. | +| Windows static plugin, real GPU | Build and tests pass | The static-plugin and internal-EP builds have matching results for their common provider tests. | | Windows shared plugin | Build and tests pass | The DLL continues to export the unprefixed entry points required by `EpLibraryPlugin`. | | Linux static plugin, GCC | Build passes | Build-only CI covers GCC and Python wheel linkage; GPU execution is not exercised. | -| Emscripten static plugin | Build passes | Build-only CI and symbol inspection verify linked entry points. A local prototype passed 2152 browser WebGPU tests, but the shipping artifacts and blocking CI still use the internal EP path. | +| Emscripten static plugin | Build passes | Build-only CI and symbol inspection verify linked entry points. A local prototype passed the browser WebGPU suite, but shipping artifacts and blocking CI still use the internal EP path. | | Minimal build | Rejected at configure time | Prevents a provider from being linked into a binary that omits its registration infrastructure. | -The adapter configuration excludes 12 operator cases that depend on internal WebGPU test infrastructure, in addition -to white-box and disabled tests. Re-registration after complete `OrtEnv` teardown and registration of more than one -static plugin EP also remain untested. +The adapter configuration excludes operator cases that depend on internal WebGPU test infrastructure, in addition +to white-box and disabled tests. + +## Appendix: implementation map + +| Decision | Owning code | +| --- | --- | +| D1 static registry | `core/session/plugin_ep/ep_static_plugins.cc`, `ep_library_static_plugin.h` | +| D1 shared factory lifecycle | `core/session/plugin_ep/ep_library_plugin.cc` | +| D2 entry point prefixing | `core/providers/webgpu/ep/api.cc`, `include/onnxruntime/ep/api.h` | +| D3 process-global ownership | `core/providers/webgpu/ep/api.cc` `ReleaseEpFactory` | +| D4 build option | `cmake/CMakeLists.txt`, `cmake/onnxruntime_providers_webgpu.cmake`, `tools/ci_build/build.py` | +| D5 optional library path | `test/unittest_util/test_dynamic_plugin_ep.cc` | +| D6 registration timing | `core/session/ort_env.cc`, `Environment::CreateAndRegisterStaticPluginEps` | +| D7 C++ API initialization | `include/onnxruntime/ep/api.h` | From 4cc65ebe429c427903fa6353f46f84db1df34a54 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Mon, 14 Sep 2026 09:33:26 -0700 Subject: [PATCH 079/107] trim comment --- .github/workflows/windows_webgpu.yml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/.github/workflows/windows_webgpu.yml b/.github/workflows/windows_webgpu.yml index ecd539938e35c..91666da32504a 100644 --- a/.github/workflows/windows_webgpu.yml +++ b/.github/workflows/windows_webgpu.yml @@ -229,9 +229,7 @@ jobs: ${{ github.workspace }}/RelWithDebInfo/RelWithDebInfo/dxil.dll # Same plugin EP sources as webgpu_plugin_build_x64_RelWithDebInfo, but linked into the ORT binary - # instead of built as a separate shared library. Registered by ORT core at environment creation. - # See docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/ - # static_plugin_ep_registration_design.md + # instead of built as a separate shared library. webgpu_static_plugin_build_x64_RelWithDebInfo: runs-on: [ "self-hosted", From f5819518f1c81bafa416ce766e5b99b0be285465 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Mon, 14 Sep 2026 09:35:41 -0700 Subject: [PATCH 080/107] fix cmake option comment --- cmake/CMakeLists.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake/CMakeLists.txt b/cmake/CMakeLists.txt index 0c14b9afecc17..da9d153e8e237 100644 --- a/cmake/CMakeLists.txt +++ b/cmake/CMakeLists.txt @@ -178,7 +178,7 @@ option(onnxruntime_USE_CANN "Build with CANN support" OFF) option(onnxruntime_USE_XNNPACK "Build with XNNPACK support. Provides an alternative math library on ARM, WebAssembly and x86." OFF) option(onnxruntime_USE_WEBNN "Build with WebNN support. Enable hardware acceleration in web browsers." OFF) option(onnxruntime_USE_WEBGPU "Build with WebGPU support. Enable WebGPU via C/C++ interface." OFF) -option(onnxruntime_USE_EP_API_ADAPTERS "Build EP (e.g. WebGPU) as a plugin EP shared library using EP API adapters" OFF) +option(onnxruntime_USE_EP_API_ADAPTERS "Build EP (e.g. WebGPU) as a plugin EP library using EP API adapters" OFF) option(onnxruntime_WEBGPU_STATIC_PLUGIN "Build the WebGPU plugin EP into the ORT binary instead of a separate shared library. Requires onnxruntime_USE_EP_API_ADAPTERS." OFF) option(onnxruntime_USE_EXTERNAL_DAWN "Build with treating Dawn as external dependency. Will not link Dawn at build time." OFF) option(onnxruntime_CUSTOM_DAWN_SRC_PATH "Path to custom Dawn src dir.") From b99d1169bb854e5e75d39ac214134be4ee583f35 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Mon, 14 Sep 2026 10:10:55 -0700 Subject: [PATCH 081/107] update GetSupportedDevices comment --- include/onnxruntime/core/session/onnxruntime_ep_c_api.h | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/include/onnxruntime/core/session/onnxruntime_ep_c_api.h b/include/onnxruntime/core/session/onnxruntime_ep_c_api.h index 9537f8b98be7e..575d98a8fb8d3 100644 --- a/include/onnxruntime/core/session/onnxruntime_ep_c_api.h +++ b/include/onnxruntime/core/session/onnxruntime_ep_c_api.h @@ -2799,9 +2799,8 @@ struct OrtEpFactory { * \snippet{doc} snippets.dox OrtStatus Return Value * * \remarks Called during execution provider library registration. The OrtEpDevice instances returned are added to - * the environment after this returns, so an implementation must not depend on the environment's registered - * execution provider devices. OrtApi::GetEpDevices will not include this factory's devices, and whether it - * includes devices from other libraries depends on registration order. + * the environment after this returns. An implementation should not depend on other OrtEpDevices returned by + * OrtApi::GetEpDevices since those depend on the execution provider library registration order. * * Environment configuration entries are populated before any library is registered, so * OrtEpApi::GetEnvConfigEntries may be called from this function. From 2f01282e69547f27b47849c0e58ff4e490f90748 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Mon, 14 Sep 2026 10:26:44 -0700 Subject: [PATCH 082/107] clarify comment for KernelInfoGetAttribute_string --- include/onnxruntime/core/session/onnxruntime_c_api.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/include/onnxruntime/core/session/onnxruntime_c_api.h b/include/onnxruntime/core/session/onnxruntime_c_api.h index 4dd086c62cf96..8c7f6e4389363 100644 --- a/include/onnxruntime/core/session/onnxruntime_c_api.h +++ b/include/onnxruntime/core/session/onnxruntime_c_api.h @@ -2394,7 +2394,9 @@ struct OrtApi { * * If the `size` parameter is less than the actual string attribute's size and `out` * is not nullptr, the value of `size` is set to the true size of the string attribute - * and a failure status is returned.) + * and a failure status is returned. + * + * The true size of the string attribute includes the trailing null character. * * \param[in] info ::OrtKernelInfo instance * \param[in] name Null terminated string of the name of the attribute From 3fd1009c54bd1928869a2f942757df470fddc8c7 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Mon, 14 Sep 2026 14:55:37 -0700 Subject: [PATCH 083/107] Clarify Emscripten WebGPU discovery in Node.js Remove the inaccurate claim that the navigator.gpu check detects WebGPU objects exposed only through globalThis. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- onnxruntime/core/platform/emscripten/device_discovery.cc | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/onnxruntime/core/platform/emscripten/device_discovery.cc b/onnxruntime/core/platform/emscripten/device_discovery.cc index c5e4d12c52ee7..fd11ce9ea00e9 100644 --- a/onnxruntime/core/platform/emscripten/device_discovery.cc +++ b/onnxruntime/core/platform/emscripten/device_discovery.cc @@ -18,9 +18,8 @@ namespace { // also the same signal ORT Web itself uses to decide whether WebGPU is usable (see initEp() in // js/web/lib/wasm/wasm-core-impl.ts). // -// navigator is defined in both window and worker scopes, so this works on either. Node.js has no native WebGPU -// implementation (nodejs/node#42896 was closed as not planned), but the userland Dawn based packages install their -// GPU object onto globalThis, so this check picks those up too. Where neither is present no GPU is reported. +// navigator is defined in both window and worker scopes, so this works on either. Hosts without navigator.gpu, +// including Node.js by default, report no GPU. bool IsWebGpuAvailable() { // clang-format off // The body below is JavaScript, not C++; clang-format would otherwise rewrite `!==` as `!= =`. From c7e06cdf95f5bbe13d1c5c4a885c9bc319de6141 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Mon, 14 Sep 2026 15:45:50 -0700 Subject: [PATCH 084/107] Address static plugin EP review feedback Use the non-throwing C API to acquire the default allocator and clarify that static EP registration calls the private non-locking overload. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- include/onnxruntime/ep/adapter/op_kernel_info.h | 10 +++++----- onnxruntime/core/session/environment.cc | 7 ++++--- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/include/onnxruntime/ep/adapter/op_kernel_info.h b/include/onnxruntime/ep/adapter/op_kernel_info.h index da2270f604fd1..ebf3c22b189f7 100644 --- a/include/onnxruntime/ep/adapter/op_kernel_info.h +++ b/include/onnxruntime/ep/adapter/op_kernel_info.h @@ -260,7 +260,8 @@ struct OpKernelInfo { } static Status GetAttrsImpl(const OrtKernelInfo* info, const char* name, std::vector& out) { - Ort::AllocatorWithDefaultOptions allocator; + OrtAllocator* allocator = nullptr; + ORT_RETURN_IF_ERROR(ToStatus(Ort::GetApi().GetAllocatorWithDefaultOptions(&allocator))); size_t size = 0; ORT_RETURN_IF_ERROR( ToStatus(Ort::GetApi().KernelInfoGetAttributeArray_string(info, name, allocator, nullptr, &size))); @@ -276,14 +277,13 @@ struct OpKernelInfo { // The allocator owns both the array and every string in it, so release them on all exit paths: constructing the // std::string copies below can throw. Free through the OrtAllocator function pointer rather than // Ort::AllocatorWithDefaultOptions::Free(), which throws on failure and so must not run in this deleter. - OrtAllocator* raw_allocator = allocator; - auto free_raw_values = [raw_allocator, size](char** values_to_free) { + auto free_raw_values = [allocator, size](char** values_to_free) { for (size_t i = 0; i < size; ++i) { if (values_to_free[i] != nullptr) { - raw_allocator->Free(raw_allocator, values_to_free[i]); + allocator->Free(allocator, values_to_free[i]); } } - raw_allocator->Free(raw_allocator, values_to_free); + allocator->Free(allocator, values_to_free); }; std::unique_ptr raw_values_guard{raw_values, std::move(free_raw_values)}; diff --git a/onnxruntime/core/session/environment.cc b/onnxruntime/core/session/environment.cc index 0c8ac93bb6238..fb8ba6186a6db 100644 --- a/onnxruntime/core/session/environment.cc +++ b/onnxruntime/core/session/environment.cc @@ -648,9 +648,10 @@ Status Environment::CreateAndRegisterInternalEps() { } Status Environment::CreateAndRegisterStaticPluginEps(StaticPluginEpRegistrationToken) { - // Intentionally does not take mutex_. That is safe because the caller holds the OrtEnv creation mutex, so no other - // thread can obtain a reference to this Environment yet. Taking mutex_ here would deadlock if a plugin EP called an - // Environment API from its OrtEpFactory::GetSupportedDevices implementation. + // Intentionally does not take mutex_ and calls the private, non-locking RegisterExecutionProviderLibrary overload. + // This is safe because the caller holds the OrtEnv creation mutex, so no other thread can obtain a reference to this + // Environment yet. Taking mutex_ here would deadlock if a plugin EP called a locking Environment API from + // OrtEpFactory::GetSupportedDevices. auto static_plugin_ep_libraries = CreateStaticPluginEpLibraries(); for (auto& ep_library : static_plugin_ep_libraries) { const std::string registration_name = ep_library->RegistrationName(); From 16fa85f84a8cd8ce62d6675cb8bcace91b2c6f56 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Tue, 15 Sep 2026 11:43:41 -0700 Subject: [PATCH 085/107] Fix WASM device discovery test failure --- onnxruntime/test/platform/device_discovery_test.cc | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/onnxruntime/test/platform/device_discovery_test.cc b/onnxruntime/test/platform/device_discovery_test.cc index 13b515697ad8c..c3a66dadaa568 100644 --- a/onnxruntime/test/platform/device_discovery_test.cc +++ b/onnxruntime/test/platform/device_discovery_test.cc @@ -33,8 +33,13 @@ TEST(DeviceDiscoveryTest, GpuDevicesHaveValidProperties) { // GPU detection should not crash. If GPUs are present, validate their properties. for (const auto& gpu_device : gpu_devices) { +#if !defined(__wasm__) EXPECT_NE(gpu_device.vendor_id, 0u); - // Note: device_id may be 0 on some platforms (e.g., Apple Silicon) where it is not populated. +#else + // WASM GPU device discovery is limited so vendor ID is unknown. + EXPECT_EQ(gpu_device.vendor_id, 0u); +#endif + // Note: gpu_device.device_id may be 0 on some platforms (e.g., Apple Silicon) where it is not populated. } } From 427d2ea84eb107265a32f7a11cd6de6689f4ada1 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Tue, 22 Sep 2026 17:31:25 -0700 Subject: [PATCH 086/107] Make plugin EP registration failure-safe Adopt plugin factories, allocators, and data transfers with RAII so partial callback outputs are released on errors. Roll back environment registrations when setup fails and add focused ownership and retry coverage. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .../static_plugin_ep_registration_design.md | 8 +- .../onnxruntime/core/session/environment.h | 2 - .../core/framework/plugin_data_transfer.cc | 6 +- .../core/framework/plugin_data_transfer.h | 29 +- onnxruntime/core/providers/webgpu/ep/api.cc | 6 +- .../core/session/allocator_adapters.cc | 5 +- onnxruntime/core/session/environment.cc | 247 ++++++++++++------ .../session/plugin_ep/ep_allocator_utils.cc | 72 +++++ .../session/plugin_ep/ep_allocator_utils.h | 35 +++ .../core/session/plugin_ep/ep_library.h | 14 +- .../session/plugin_ep/ep_library_internal.h | 16 +- .../session/plugin_ep/ep_library_plugin.cc | 4 +- .../session/plugin_ep/ep_library_plugin.h | 15 +- .../plugin_ep/ep_library_plugin_utils.cc | 90 ++++--- .../plugin_ep/ep_library_plugin_utils.h | 30 +-- .../plugin_ep/ep_library_provider_bridge.cc | 4 +- .../plugin_ep/ep_library_provider_bridge.h | 13 +- .../plugin_ep/ep_library_static_plugin.cc | 11 +- .../plugin_ep/ep_library_static_plugin.h | 15 +- .../ep_plugin_provider_interfaces.cc | 51 +--- .../library/example_plugin_ep/ep_factory.cc | 5 + .../example_plugin_ep/ep_test_hooks.cc | 7 + .../library/example_plugin_ep/ep_test_hooks.h | 2 + .../example_plugin_ep_library.def | 1 - .../example_plugin_ep_library.lds | 1 + onnxruntime/test/autoep/test_autoep_utils.cc | 4 + onnxruntime/test/autoep/test_autoep_utils.h | 2 + onnxruntime/test/autoep/test_registration.cc | 32 +++ .../framework/ep_library_plugin_utils_test.cc | 110 ++++++++ .../test/framework/ep_plugin_provider_test.cc | 156 +++++++++++ 30 files changed, 751 insertions(+), 242 deletions(-) create mode 100644 onnxruntime/core/session/plugin_ep/ep_allocator_utils.cc create mode 100644 onnxruntime/core/session/plugin_ep/ep_allocator_utils.h create mode 100644 onnxruntime/test/framework/ep_library_plugin_utils_test.cc diff --git a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md index d5c4d9ebb7405..9bd889abf640d 100644 --- a/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md +++ b/docs/design/webgpu_ep_extraction/plugin_boundary_and_web_integration/static_plugin_ep_registration_design.md @@ -58,9 +58,11 @@ flowchart LR ### Failure handling `EpLibraryStaticPlugin` releases factories created before `CreateEpFactories` reports a failure. Existing -`Environment::RegisterExecutionProviderLibrary` diagnostics cover duplicate names and factory or device errors. If -automatic registration fails, `OrtEnv::GetOrCreateInstance` unpublishes and destroys the partial singleton before -returning the error. Other threads therefore cannot observe incomplete static registration. +`Environment::RegisterExecutionProviderLibrary` diagnostics cover duplicate names and factory or device errors. +Registration stages factories, devices, allocators, and data transfers and rolls back dependent objects before +releasing their factory on failure. If automatic registration fails, `OrtEnv::GetOrCreateInstance` unpublishes and +destroys the partial singleton before returning the error. Other threads therefore cannot observe incomplete static +registration. ## Decisions diff --git a/include/onnxruntime/core/session/environment.h b/include/onnxruntime/core/session/environment.h index a8bb00fc1ef3d..6033c6e4edf13 100644 --- a/include/onnxruntime/core/session/environment.h +++ b/include/onnxruntime/core/session/environment.h @@ -276,8 +276,6 @@ class Environment { // providing a CPU allocator. std::unique_ptr default_cpu_ort_allocator_; - using OrtAllocatorUniquePtr = std::unique_ptr>; - #if !defined(ORT_MINIMAL_BUILD) // register EPs that are built into the ORT binary so they can take part in AutoEP selection // added to ep_libraries diff --git a/onnxruntime/core/framework/plugin_data_transfer.cc b/onnxruntime/core/framework/plugin_data_transfer.cc index d6b1680176815..9de3ed7bf8528 100644 --- a/onnxruntime/core/framework/plugin_data_transfer.cc +++ b/onnxruntime/core/framework/plugin_data_transfer.cc @@ -44,8 +44,8 @@ Status DataTransfer::CopyTensors(const std::vector& src_dst_pairs) c streams.push_back(reinterpret_cast(src_dst_pairs[i].src_stream)); } - auto* status = impl_.CopyTensors(&impl_, src_values.data(), dst_values.data(), streams.data(), - src_dst_pairs.size()); + auto* status = impl_->CopyTensors(impl_.get(), src_values.data(), dst_values.data(), streams.data(), + src_dst_pairs.size()); return ToStatusAndRelease(status); } @@ -59,7 +59,7 @@ Status DataTransfer::CopyTensorImpl(const Tensor& src_tensor, Tensor& dst_tensor const OrtValue* src_ptr = &src; OrtValue* dst_ptr = &dst; OrtSyncStream* stream_ptr = nullptr; // static_cast(stream); - auto* status = impl_.CopyTensors(&impl_, &src_ptr, &dst_ptr, &stream_ptr, 1); + auto* status = impl_->CopyTensors(impl_.get(), &src_ptr, &dst_ptr, &stream_ptr, 1); return ToStatusAndRelease(status); } diff --git a/onnxruntime/core/framework/plugin_data_transfer.h b/onnxruntime/core/framework/plugin_data_transfer.h index e8ad29f4c0609..f9be30cca2703 100644 --- a/onnxruntime/core/framework/plugin_data_transfer.h +++ b/onnxruntime/core/framework/plugin_data_transfer.h @@ -2,6 +2,9 @@ // Licensed under the MIT License. #pragma once +#include +#include + #include "core/framework/data_transfer.h" #include "core/framework/error_code_helper.h" #include "core/framework/ort_value.h" @@ -11,21 +14,35 @@ namespace onnxruntime { namespace plugin_ep { +struct OrtDataTransferImplDeleter { + void operator()(OrtDataTransferImpl* data_transfer) const noexcept { + if (data_transfer != nullptr && data_transfer->Release != nullptr) { + data_transfer->Release(data_transfer); + } + } +}; + +using OrtDataTransferImplPtr = std::unique_ptr; + /// /// Class to implement IDataTransfer for plugin execution providers. /// It uses the OrtDataTransferImpl from the plugin EP factory to implement the data transfer functionality. /// class DataTransfer : public IDataTransfer { public: - DataTransfer(OrtDataTransferImpl& impl) - : impl_{impl} { + explicit DataTransfer(OrtDataTransferImplPtr impl) + : impl_{std::move(impl)} { + ORT_ENFORCE(impl_ != nullptr, "OrtDataTransferImpl must not be null."); + ORT_ENFORCE(impl_->Release != nullptr, "OrtDataTransferImpl must provide a Release function."); + ORT_ENFORCE(impl_->CanCopy != nullptr, "OrtDataTransferImpl must provide a CanCopy function."); + ORT_ENFORCE(impl_->CopyTensors != nullptr, "OrtDataTransferImpl must provide a CopyTensors function."); } bool CanCopy(const OrtDevice& src_device, const OrtDevice& dst_device) const override { const OrtMemoryDevice* src_memory_device = static_cast(&src_device); const OrtMemoryDevice* dst_memory_device = static_cast(&dst_device); - return impl_.CanCopy(&impl_, src_memory_device, dst_memory_device); + return impl_->CanCopy(impl_.get(), src_memory_device, dst_memory_device); } Status CopyTensor(const Tensor& src, Tensor& dst) const override { @@ -38,14 +55,10 @@ class DataTransfer : public IDataTransfer { Status CopyTensors(const std::vector& src_dst_pairs) const override; - ~DataTransfer() override { - impl_.Release(&impl_); - } - private: Status CopyTensorImpl(const Tensor& src, Tensor& dst, onnxruntime::Stream* stream = nullptr) const; - OrtDataTransferImpl& impl_; + OrtDataTransferImplPtr impl_; }; } // namespace plugin_ep } // namespace onnxruntime diff --git a/onnxruntime/core/providers/webgpu/ep/api.cc b/onnxruntime/core/providers/webgpu/ep/api.cc index 5447f19aa0368..7f77c20f4f124 100644 --- a/onnxruntime/core/providers/webgpu/ep/api.cc +++ b/onnxruntime/core/providers/webgpu/ep/api.cc @@ -13,9 +13,9 @@ #include "core/session/onnxruntime_env_config_keys.h" // When this EP is statically linked into the host binary instead of being built as a separate plugin library, the -// entry points below are given a `WebGpu_` prefix so that multiple statically linked plugin EPs can coexist in one -// binary. ORT core declares the prefixed names in onnxruntime/core/session/plugin_ep/ep_static_plugins.cc, which -// spells them out literally, so the prefix is hardcoded on both sides rather than configured by the build. +// entry points below are given a `WebGpu_` prefix. ORT core declares the prefixed names in +// onnxruntime/core/session/plugin_ep/ep_static_plugins.cc, which spells them out literally, so the prefix is hardcoded +// on both sides rather than configured by the build. // // The shared library build must leave the entry points unprefixed, because they are resolved by exact name. #if defined(ORT_PLUGIN_EP_STATICALLY_LINKED) diff --git a/onnxruntime/core/session/allocator_adapters.cc b/onnxruntime/core/session/allocator_adapters.cc index 62316dd737bf1..84d0641d9a717 100644 --- a/onnxruntime/core/session/allocator_adapters.cc +++ b/onnxruntime/core/session/allocator_adapters.cc @@ -139,9 +139,8 @@ onnxruntime::AllocatorPtr OrtAllocatorImplWrappingIAllocator::GetWrappedIAllocat IAllocatorImplWrappingOrtAllocator::IAllocatorImplWrappingOrtAllocator(OrtAllocator* ort_allocator) : IAllocator(*ort_allocator->Info(ort_allocator)) { - ort_allocator_ = OrtAllocatorUniquePtr(ort_allocator, [](OrtAllocator*) { - // no-op - }); + void (*noop_deleter)(OrtAllocator*) = [](OrtAllocator*) {}; + ort_allocator_ = OrtAllocatorUniquePtr(ort_allocator, noop_deleter); } IAllocatorImplWrappingOrtAllocator::IAllocatorImplWrappingOrtAllocator(OrtAllocatorUniquePtr ort_allocator) diff --git a/onnxruntime/core/session/environment.cc b/onnxruntime/core/session/environment.cc index fb8ba6186a6db..cb9565dae7070 100644 --- a/onnxruntime/core/session/environment.cc +++ b/onnxruntime/core/session/environment.cc @@ -3,9 +3,12 @@ #include "core/session/environment.h" +#include #include +#include #include "core/common/basic_types.h" +#include "core/common/safeint.h" #include "core/framework/allocator.h" #include "core/framework/allocator_utils.h" #include "core/framework/error_code_helper.h" @@ -18,6 +21,7 @@ #include "core/session/inference_session.h" #include "core/session/onnxruntime_env_config_keys.h" #include "core/session/onnxruntime_ep_device_ep_metadata_keys.h" +#include "core/session/plugin_ep/ep_allocator_utils.h" #include "core/session/plugin_ep/ep_factory_internal.h" #include "core/session/plugin_ep/ep_library_internal.h" #include "core/session/plugin_ep/ep_library_plugin.h" @@ -542,14 +546,16 @@ namespace { Status CreateDataTransferForFactory(OrtEpFactory& ep_factory, std::unique_ptr& data_transfer) { OrtDataTransferImpl* data_transfer_impl = nullptr; - OrtStatus* status = ep_factory.CreateDataTransfer(&ep_factory, &data_transfer_impl); - if (status != nullptr) { + OrtStatus* ort_status = ep_factory.CreateDataTransfer(&ep_factory, &data_transfer_impl); + plugin_ep::OrtDataTransferImplPtr owned_impl{data_transfer_impl}; + + if (ort_status != nullptr) { return ORT_MAKE_STATUS(ONNXRUNTIME, EP_FAIL, - "Error creating data transfer: ", ToStatusAndRelease(status).ToString()); + "Error creating data transfer: ", ToStatusAndRelease(ort_status).ToString()); } - if (data_transfer_impl != nullptr) { - data_transfer = std::make_unique(*data_transfer_impl); + if (owned_impl != nullptr) { + data_transfer = std::make_unique(std::move(owned_impl)); } return Status::OK(); @@ -570,53 +576,132 @@ Status Environment::RegisterExecutionProviderLibrary(const std::string& registra const Env& env = Env::Default(); env.GetTelemetryProvider().LogRegisterEpLibraryStart(registration_name); - if (ep_libraries_.count(registration_name) > 0) { - auto status = ORT_MAKE_STATUS(ONNXRUNTIME, FAIL, "library is already registered under ", registration_name); - env.GetTelemetryProvider().LogRegisterEpLibraryEnd(registration_name, status); - return status; - } - auto status = Status::OK(); ORT_TRY { - // create the EpInfo which loads the library if required - std::unique_ptr ep_info = nullptr; - ORT_RETURN_IF_ERROR(EpInfo::Create(std::move(ep_library), ep_info)); + status = [&]() -> Status { + bool registration_succeeded = false; - // add the pointers to the OrtEpDevice instances to our global list - execution_devices_.reserve(execution_devices_.size() + ep_info->execution_devices.size()); - for (const auto& ed : ep_info->execution_devices) { - execution_devices_.push_back(ed.get()); - - // add shared allocators so they're available without an inference session being required. - // we don't replace an existing allocator as we just need one to exist for the OrtMemoryInfo and we don't want - // to blow away any custom allocators previously added by the user. - if (ed->device_memory_info != nullptr) { - ORT_RETURN_IF_ERROR(CreateSharedAllocatorImpl(*ed, *ed->device_memory_info, OrtDeviceAllocator, nullptr, - nullptr, /*replace_existing*/ false)); + if (ep_libraries_.contains(registration_name)) { + return ORT_MAKE_STATUS(ONNXRUNTIME, FAIL, "EP library is already registered under ", registration_name); } - if (ed->host_accessible_memory_info != nullptr) { - ORT_RETURN_IF_ERROR(CreateSharedAllocatorImpl(*ed, *ed->host_accessible_memory_info, OrtDeviceAllocator, - nullptr, nullptr, /*replace_existing*/ false)); + std::unique_ptr ep_info; + ORT_RETURN_IF_ERROR(EpInfo::Create(std::move(ep_library), ep_info, internal_factories)); + ORT_RETURN_IF_NOT(ep_info != nullptr); + + // Register shared allocators + const size_t max_allocator_count = SafeInt(ep_info->execution_devices.size()) * 2; + std::vector registered_allocator_memory_infos; + registered_allocator_memory_infos.reserve(max_allocator_count); + + auto rollback_shared_allocator_registrations = gsl::finally([&]() { + if (registration_succeeded) { + return; + } + for (auto it = registered_allocator_memory_infos.rbegin(); it != registered_allocator_memory_infos.rend(); ++it) { + ORT_IGNORE_RETURN_VALUE(UnregisterAllocatorImpl(**it, /*error_if_not_found*/ false)); + } + }); + + for (const auto& ed : ep_info->execution_devices) { + // Add shared allocators so they're available without an inference session being required. Do not replace an + // existing allocator: we only need one for the OrtMemoryInfo and must preserve user-provided allocators. + for (const OrtMemoryInfo* memory_info : {ed->device_memory_info, ed->host_accessible_memory_info}) { + if (memory_info == nullptr) { + continue; + } + + OrtAllocator* registered_allocator = nullptr; + ORT_RETURN_IF_ERROR(CreateSharedAllocatorImpl(*ed, *memory_info, OrtDeviceAllocator, nullptr, + ®istered_allocator, /*replace_existing*/ false)); + if (registered_allocator != nullptr) { + registered_allocator_memory_infos.push_back(memory_info); + } + } } - } - for (auto* factory : ep_info->factories) { - std::unique_ptr data_transfer; - ORT_RETURN_IF_ERROR(CreateDataTransferForFactory(*factory, data_transfer)); + // Register data transfers + std::vector registered_data_transfers; + registered_data_transfers.reserve(ep_info->factories.size()); + ep_info->data_transfers.reserve(ep_info->factories.size()); - if (data_transfer) { - ep_info->data_transfers.push_back(data_transfer.get()); // store so we can unregister in the unload - ORT_RETURN_IF_ERROR(data_transfer_mgr_.RegisterDataTransfer(std::move(data_transfer))); + auto rollback_data_transfer_registrations = gsl::finally([&]() { + if (registration_succeeded) { + return; + } + + for (auto it = registered_data_transfers.rbegin(); it != registered_data_transfers.rend(); ++it) { + ORT_IGNORE_RETURN_VALUE(data_transfer_mgr_.UnregisterDataTransfer(*it)); + } + }); + + for (auto* factory : ep_info->factories) { + std::unique_ptr data_transfer; + ORT_RETURN_IF_ERROR(CreateDataTransferForFactory(*factory, data_transfer)); + + if (data_transfer) { + auto* data_transfer_ptr = data_transfer.get(); + ORT_RETURN_IF_ERROR(data_transfer_mgr_.RegisterDataTransfer(std::move(data_transfer))); + registered_data_transfers.push_back(data_transfer_ptr); + ep_info->data_transfers.push_back(data_transfer_ptr); + } } - } - for (const auto& internal_factory : internal_factories) { - internal_ep_factories_.insert(internal_factory); - } + // Register internal factories + std::vector registered_internal_factories; + registered_internal_factories.reserve(internal_factories.size()); + + auto rollback_internal_factory_registration = gsl::finally([&]() { + if (registration_succeeded) { + return; + } - ep_libraries_[registration_name] = std::move(ep_info); + for (auto it = registered_internal_factories.rbegin(); it != registered_internal_factories.rend(); ++it) { + internal_ep_factories_.erase(*it); + } + }); + + for (auto* internal_factory : internal_factories) { + if (internal_ep_factories_.insert(internal_factory).second) { + registered_internal_factories.push_back(internal_factory); + } + } + + // Register EP devices + auto rollback_execution_devices_registration = gsl::finally( + [&, original_size = execution_devices_.size()]() { + if (registration_succeeded) { + return; + } + + execution_devices_.erase(execution_devices_.begin() + original_size, + execution_devices_.end()); + }); + + execution_devices_.reserve(SafeInt(execution_devices_.size()) + + ep_info->execution_devices.size()); + std::transform(ep_info->execution_devices.begin(), ep_info->execution_devices.end(), + std::back_inserter(execution_devices_), + [](const std::unique_ptr& ep_device) { + return ep_device.get(); + }); + + // Finally, register the EP library + + // Notes: + // `ep_info`'s target must stay alive for any rollback logic that may need to run. + // Directly moving into `ep_libraries_.emplace()` could cause it to be prematurely destroyed if emplace throws + // after ownership has transferred from `ep_info`. This is why we first emplace an empty unique_ptr and then + // swap into it. + // + // We ignore `emplace_result.second` since we already checked for an existing entry in `ep_libraries_` above. + const auto emplace_result = ep_libraries_.emplace(registration_name, nullptr); + emplace_result.first->second.swap(ep_info); + + registration_succeeded = true; + return Status::OK(); + }(); } ORT_CATCH(const std::exception& ex) { ORT_HANDLE_EXCEPTION([&]() { @@ -862,66 +947,52 @@ Status Environment::CreateSharedAllocatorImpl(const OrtEpDevice& ep_device, "any arena options via the allocator options."); } - // we need to remove from shared_ort_allocators_ first in case the entry in shared_allocators_ owns the pointer in - // shared_ort_allocators_. - if (auto it = FindExistingAllocator(shared_ort_allocators_, memory_info, /*match_name*/ true); - it != shared_ort_allocators_.end()) { - if (!replace_existing) { - LOGS_DEFAULT(INFO) << "A shared allocator is already registered for " << memory_info - << ". Skipping EP allocator creation."; - return Status::OK(); - } - - shared_ort_allocators_.erase(it); - } - // we only want one shared allocator for an OrtDevice in the shared_allocators_ so that it's deterministic which // one will be used for an inference session. ignore the name so that is the case. - if (auto it = FindExistingAllocator(shared_allocators_, memory_info, /*match_name*/ false); - it != shared_allocators_.end()) { - if (!replace_existing) { - LOGS_DEFAULT(INFO) << "A shared allocator is already registered for " << memory_info - << ". Skipping EP allocator creation."; - return Status::OK(); + auto existing_ort_allocator = + FindExistingAllocator(shared_ort_allocators_, memory_info, /*match_name*/ true); + + auto existing_shared_allocator = + FindExistingAllocator(shared_allocators_, memory_info, /*match_name*/ false); + + if (!replace_existing && + (existing_ort_allocator != shared_ort_allocators_.end() || + existing_shared_allocator != shared_allocators_.end())) { + if (allocator_out != nullptr) { + *allocator_out = nullptr; } - shared_allocators_.erase(it); + LOGS_DEFAULT(INFO) << "A shared allocator is already registered for " << memory_info + << ". Skipping EP allocator creation."; + return Status::OK(); } - OrtAllocator* allocator = nullptr; - auto* ort_status = ep_device.ep_factory->CreateAllocator(ep_device.ep_factory, &memory_info, allocator_options, - &allocator); - if (ort_status != nullptr) { - return ToStatusAndRelease(ort_status); + // we need to remove from shared_ort_allocators_ first in case the entry in shared_allocators_ owns the pointer in + // shared_ort_allocators_. + if (existing_ort_allocator != shared_ort_allocators_.end()) { + shared_ort_allocators_.erase(existing_ort_allocator); } - if (allocator->Info(allocator)->alloc_type == OrtAllocatorType::OrtArenaAllocator) { - return ORT_MAKE_STATUS(ONNXRUNTIME, FAIL, - "OrtEpFactory returned an allocator with OrtAllocatorType of OrtArenaAllocator. " - "This type is reserved for ONNX Runtime internal usage only, as any arena usage by the " - "EP library should be opaque to ORT"); + if (existing_shared_allocator != shared_allocators_.end()) { + shared_allocators_.erase(existing_shared_allocator); } - auto* ep_factory = ep_device.ep_factory; - auto ort_allocator = OrtAllocatorUniquePtr(allocator, - [ep_factory](OrtAllocator* allocator) { - ep_factory->ReleaseAllocator(ep_factory, allocator); - }); - - shared_ort_allocators_.insert(allocator); + // Create a new allocator. - // Wrap as IArena when the plugin allocator implements Shrink(), making it - // discoverable by session-level arena management (e.g. ShrinkMemoryArenas). + auto* ep_factory = ep_device.ep_factory; + OrtAllocator* raw_ort_allocator = nullptr; AllocatorPtr shared_allocator; - if (allocator->version >= 25 && allocator->Shrink != nullptr) { - shared_allocator = std::make_shared(std::move(ort_allocator)); - } else { - shared_allocator = std::make_shared(std::move(ort_allocator)); - } + ORT_RETURN_IF_ERROR(ep_allocator_utils::CreateAndWrapEpAllocator(*ep_factory, memory_info, allocator_options, + shared_allocator, &raw_ort_allocator)); + ORT_RETURN_IF_NOT(raw_ort_allocator != nullptr, "OrtEpFactory returned a null allocator."); + + shared_allocators_.reserve(SafeInt(shared_allocators_.size()) + 1); + const auto [_, inserted] = shared_ort_allocators_.insert(raw_ort_allocator); + ORT_RETURN_IF_NOT(inserted, "OrtEpFactory returned an allocator that is already registered."); shared_allocators_.push_back(std::move(shared_allocator)); if (allocator_out != nullptr) { - *allocator_out = allocator; + *allocator_out = raw_ort_allocator; } return Status::OK(); @@ -951,7 +1022,11 @@ Status Environment::EpInfo::Create(std::unique_ptr library_in, std::u instance.internal_factories = internal_factories; ORT_RETURN_IF_ERROR(instance.library->Load()); - instance.factories = instance.library->GetFactories(); + + { + auto factories = instance.library->GetFactories(); + instance.factories.assign(factories.begin(), factories.end()); + } // OrtHardwareDevice instances to pass to GetSupportedDevices. const auto& sorted_devices = GetSortedHardwareDevices(); diff --git a/onnxruntime/core/session/plugin_ep/ep_allocator_utils.cc b/onnxruntime/core/session/plugin_ep/ep_allocator_utils.cc new file mode 100644 index 0000000000000..2e501c21ee834 --- /dev/null +++ b/onnxruntime/core/session/plugin_ep/ep_allocator_utils.cc @@ -0,0 +1,72 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +#include "core/session/plugin_ep/ep_allocator_utils.h" + +#include +#include +#include + +#include "core/common/common.h" +#include "core/framework/error_code_helper.h" +#include "core/session/allocator_adapters.h" + +namespace onnxruntime { +namespace ep_allocator_utils { + +Status CreateAndWrapEpAllocator(OrtEp* ep, + OrtEpFactory& ep_factory, + const OrtMemoryInfo& memory_info, + const OrtKeyValuePairs* ep_factory_allocator_options, + AllocatorPtr& allocator_out, + OrtAllocator** raw_allocator_out) { + ORT_RETURN_IF_NOT(ep_factory.CreateAllocator != nullptr && ep_factory.ReleaseAllocator != nullptr, + "OrtEpFactory must implement CreateAllocator and ReleaseAllocator."); + + const bool create_with_ep = ep != nullptr && ep->CreateAllocator != nullptr; + + std::function allocator_deleter = [&ep_factory](OrtAllocator* allocator) { + ep_factory.ReleaseAllocator(&ep_factory, allocator); + }; + + OrtAllocator* raw_allocator = nullptr; + OrtStatus* creation_ort_status = + create_with_ep + ? ep->CreateAllocator(ep, &memory_info, &raw_allocator) + : ep_factory.CreateAllocator(&ep_factory, &memory_info, ep_factory_allocator_options, &raw_allocator); + OrtAllocatorUniquePtr owned_allocator{raw_allocator, std::move(allocator_deleter)}; + + if (creation_ort_status != nullptr) { + return ToStatusAndRelease(creation_ort_status); + } + + AllocatorPtr wrapped_allocator; + if (owned_allocator != nullptr) { + const std::string_view creator = create_with_ep ? "OrtEp" : "OrtEpFactory"; + + const OrtMemoryInfo* allocator_memory_info = owned_allocator->Info(owned_allocator.get()); + ORT_RETURN_IF_NOT(allocator_memory_info != nullptr, creator, " returned an allocator with null memory info."); + + ORT_RETURN_IF_NOT(allocator_memory_info->alloc_type != OrtAllocatorType::OrtArenaAllocator, + creator, + " returned an allocator with OrtAllocatorType of OrtArenaAllocator. " + "This type is reserved for ONNX Runtime internal usage only, as any arena usage by the " + "EP library should be opaque to ORT"); + + constexpr uint32_t kOrtAllocatorShrinkMinVersion = 25; + if (owned_allocator->version >= kOrtAllocatorShrinkMinVersion && owned_allocator->Shrink != nullptr) { + wrapped_allocator = std::make_shared(std::move(owned_allocator)); + } else { + wrapped_allocator = std::make_shared(std::move(owned_allocator)); + } + } + + if (raw_allocator_out != nullptr) { + *raw_allocator_out = raw_allocator; + } + allocator_out = std::move(wrapped_allocator); + return Status::OK(); +} + +} // namespace ep_allocator_utils +} // namespace onnxruntime diff --git a/onnxruntime/core/session/plugin_ep/ep_allocator_utils.h b/onnxruntime/core/session/plugin_ep/ep_allocator_utils.h new file mode 100644 index 0000000000000..6785509d71608 --- /dev/null +++ b/onnxruntime/core/session/plugin_ep/ep_allocator_utils.h @@ -0,0 +1,35 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +#pragma once + +#include "core/common/status.h" +#include "core/framework/allocator.h" +#include "core/session/onnxruntime_c_api.h" + +namespace onnxruntime { +namespace ep_allocator_utils { + +// Creates a plugin EP allocator using OrtEp::CreateAllocator when available, otherwise OrtEpFactory::CreateAllocator, +// and wraps the result as an IAllocator. +// Outputs are modified only on success. +Status CreateAndWrapEpAllocator(OrtEp* ep, + OrtEpFactory& ep_factory, + const OrtMemoryInfo& memory_info, + const OrtKeyValuePairs* ep_factory_allocator_options, + AllocatorPtr& allocator_out, + OrtAllocator** raw_allocator_out = nullptr); + +// Creates a plugin EP allocator using OrtEpFactory::CreateAllocator and wraps the result as an IAllocator. +// Outputs are modified only on success. +inline Status CreateAndWrapEpAllocator(OrtEpFactory& ep_factory, + const OrtMemoryInfo& memory_info, + const OrtKeyValuePairs* ep_factory_allocator_options, + AllocatorPtr& allocator_out, + OrtAllocator** raw_allocator_out = nullptr) { + return CreateAndWrapEpAllocator(/*ep*/ nullptr, ep_factory, memory_info, ep_factory_allocator_options, + allocator_out, raw_allocator_out); +} + +} // namespace ep_allocator_utils +} // namespace onnxruntime diff --git a/onnxruntime/core/session/plugin_ep/ep_library.h b/onnxruntime/core/session/plugin_ep/ep_library.h index fed9eb072c704..d99b9ca008069 100644 --- a/onnxruntime/core/session/plugin_ep/ep_library.h +++ b/onnxruntime/core/session/plugin_ep/ep_library.h @@ -3,6 +3,8 @@ #pragma once +#include +#include #include #include "core/common/status.h" #include "core/framework/provider_options.h" @@ -10,7 +12,6 @@ #include "core/session/onnxruntime_c_api.h" namespace onnxruntime { - /// /// EpLibrary is the base class for implementing support for execution provider libraries that provide /// OrtEpFactory instances. @@ -18,15 +19,22 @@ namespace onnxruntime { class EpLibrary { public: EpLibrary() = default; + virtual ~EpLibrary() = default; virtual const char* RegistrationName() const = 0; virtual const std::filesystem::path* LibraryPath() const { return nullptr; } virtual Status Load() { return Status::OK(); } - virtual const std::vector& GetFactories() = 0; // valid after Load() virtual Status Unload() { return Status::OK(); } - virtual ~EpLibrary() = default; + std::ranges::view auto GetFactories() const { + return std::views::iota(size_t{0}, GetFactoryCount()) | + std::views::transform([this](size_t index) { return GetFactory(index); }); + } ORT_DISALLOW_COPY_AND_ASSIGNMENT(EpLibrary); + + private: + virtual size_t GetFactoryCount() const = 0; + virtual OrtEpFactory* GetFactory(size_t index) const = 0; }; } // namespace onnxruntime diff --git a/onnxruntime/core/session/plugin_ep/ep_library_internal.h b/onnxruntime/core/session/plugin_ep/ep_library_internal.h index 352dba9928021..529da87bd49c6 100644 --- a/onnxruntime/core/session/plugin_ep/ep_library_internal.h +++ b/onnxruntime/core/session/plugin_ep/ep_library_internal.h @@ -21,17 +21,13 @@ namespace onnxruntime { class EpLibraryInternal : public EpLibrary { public: EpLibraryInternal(std::unique_ptr factory) - : factory_{std::move(factory)}, factory_ptrs_{factory_.get()} { + : factory_{std::move(factory)} { } const char* RegistrationName() const override { return factory_->GetName(); // same as EP name for internally registered libraries } - const std::vector& GetFactories() override { - return factory_ptrs_; - } - // there's only ever one currently EpFactoryInternal& GetInternalFactory() { return *factory_; @@ -44,6 +40,15 @@ class EpLibraryInternal : public EpLibrary { static std::vector> CreateInternalEps(bool allow_virtual_devices); private: + size_t GetFactoryCount() const override { + return 1; + } + + OrtEpFactory* GetFactory(size_t index) const override { + ORT_ENFORCE(index == 0); + return factory_.get(); + } + static std::unique_ptr CreateCpuEp(); #if defined(USE_DML) static std::unique_ptr CreateDmlEp(); @@ -53,7 +58,6 @@ class EpLibraryInternal : public EpLibrary { #endif std::unique_ptr factory_; // all internal EPs register a single factory currently - std::vector factory_ptrs_; // for convenience }; } // namespace onnxruntime diff --git a/onnxruntime/core/session/plugin_ep/ep_library_plugin.cc b/onnxruntime/core/session/plugin_ep/ep_library_plugin.cc index 3e4487fc5c5d1..aebc5c77a407b 100644 --- a/onnxruntime/core/session/plugin_ep/ep_library_plugin.cc +++ b/onnxruntime/core/session/plugin_ep/ep_library_plugin.cc @@ -28,7 +28,7 @@ Status EpLibraryPlugin::Load() { ORT_RETURN_IF_ERROR(Env::Default().GetSymbolFromLibrary(handle_, "ReleaseEpFactory", reinterpret_cast(&release_fn_))); - return ep_library_plugin_utils::CreateFactories(create_fn_, registration_name_, factories_); + return ep_library_plugin_utils::CreateFactories(create_fn_, release_fn_, registration_name_, factories_); }(); } } @@ -59,7 +59,7 @@ Status EpLibraryPlugin::Unload() { // Call ReleaseEpFactory for all factories and unload the library. // Current implementation assumes any error is permanent so does not leave pieces around to re-attempt Unload. if (handle_) { - ep_library_plugin_utils::ReleaseFactories(release_fn_, factories_, library_path_.string()); + factories_.clear(); ORT_RETURN_IF_ERROR(Env::Default().UnloadDynamicLibrary(handle_)); } diff --git a/onnxruntime/core/session/plugin_ep/ep_library_plugin.h b/onnxruntime/core/session/plugin_ep/ep_library_plugin.h index cce52b3d5d282..501c887c3f663 100644 --- a/onnxruntime/core/session/plugin_ep/ep_library_plugin.h +++ b/onnxruntime/core/session/plugin_ep/ep_library_plugin.h @@ -7,6 +7,7 @@ #include #include "core/session/plugin_ep/ep_library.h" +#include "core/session/plugin_ep/ep_library_plugin_utils.h" namespace onnxruntime { /// @@ -31,20 +32,24 @@ class EpLibraryPlugin : public EpLibrary { Status Load() override; - const std::vector& GetFactories() override { - return factories_; - } - Status Unload() override; ORT_DISALLOW_COPY_AND_ASSIGNMENT(EpLibraryPlugin); private: + size_t GetFactoryCount() const override { + return factories_.size(); + } + + OrtEpFactory* GetFactory(size_t index) const override { + return factories_[index].get(); + } + std::mutex mutex_; const std::string registration_name_; const std::filesystem::path library_path_; void* handle_{}; - std::vector factories_{}; + std::vector factories_{}; CreateEpApiFactoriesFn create_fn_{nullptr}; ReleaseEpApiFactoryFn release_fn_{nullptr}; }; diff --git a/onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.cc b/onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.cc index 10f4c021fb528..f7e62649b1bad 100644 --- a/onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.cc +++ b/onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.cc @@ -3,64 +3,72 @@ #include "core/session/plugin_ep/ep_library_plugin_utils.h" +#include +#include + +#include "core/common/common.h" #include "core/common/logging/logging.h" +#include "core/common/safeint.h" #include "core/framework/error_code_helper.h" namespace onnxruntime { namespace ep_library_plugin_utils { -Status CreateFactories(CreateEpApiFactoriesFn create_fn, const std::string& registration_name, - std::vector& factories) { - // allocate buffer for EP to add factories to. library can add up to 4 factories. - std::vector new_factories{4, nullptr}; - - size_t num_factories = 0; - ORT_RETURN_IF_ERROR(ToStatusAndRelease(create_fn(registration_name.c_str(), OrtGetApiBase(), - logging::LoggingManager::DefaultLogger().ToExternal(), - new_factories.data(), new_factories.size(), &num_factories))); - - factories.reserve(factories.size() + num_factories); - for (size_t i = 0; i < num_factories; ++i) { - factories.push_back(new_factories[i]); +void OrtEpFactoryDeleter::operator()(OrtEpFactory* factory) const noexcept { + if (factory == nullptr) { + return; } - return Status::OK(); + auto status = ToStatusAndRelease(release_fn(factory)); + if (!status.IsOK()) { + LOGS_DEFAULT(ERROR) << "ReleaseEpFactory failed with error: " << status.ErrorMessage(); + } } -void ReleaseFactories(ReleaseEpApiFactoryFn release_fn, std::vector& factories, - std::string_view library_description) { - if (factories.empty()) { - return; - } +Status CreateFactories(CreateEpApiFactoriesFn create_fn, ReleaseEpApiFactoryFn release_fn, + const std::string& registration_name, + std::vector& factories) { + constexpr size_t kMaxFactories = 4; + std::array raw_factories{}; + std::array new_factories{}; - try { - for (size_t idx = 0, end = factories.size(); idx < end; ++idx) { - auto* factory = factories[idx]; - if (factory == nullptr) { - continue; + // Make adoption non-throwing before provider code transfers any ownership to us. + const auto adopt_factories = [&]() noexcept { + for (size_t i = 0; i < raw_factories.size(); ++i) { + if (raw_factories[i] != nullptr) { + new_factories[i] = OrtEpFactoryUniquePtr{raw_factories[i], OrtEpFactoryDeleter{release_fn}}; + raw_factories[i] = nullptr; } - - auto status = ToStatusAndRelease(release_fn(factory)); - if (!status.IsOK()) { - // log it and treat it as released - LOGS_DEFAULT(ERROR) << "ReleaseEpFactory failed for: " << library_description << " with error: " - << status.ErrorMessage(); - } - - factories[idx] = nullptr; // clear the pointer in case there's a failure before all are released } + }; - factories.clear(); - } catch (const std::exception& ex) { - LOGS_DEFAULT(ERROR) << "Failed releasing EP factories from " << library_description << ": " << ex.what(); + size_t num_factories = 0; + OrtStatus* ort_status = create_fn(registration_name.c_str(), OrtGetApiBase(), + logging::LoggingManager::DefaultLogger().ToExternal(), + raw_factories.data(), raw_factories.size(), &num_factories); + adopt_factories(); + + if (ort_status != nullptr) { + return ToStatusAndRelease(ort_status); } - // TODO: Is there a better way? Is it worth worrying about? - if (!factories.empty()) { - LOGS_DEFAULT(ERROR) << "Unloading " << library_description << ". " << factories.size() - << " factories were not released due to errors. This may cause memory leaks. " - "Please check the error details in the log."; + ORT_RETURN_IF_NOT(num_factories <= new_factories.size(), + "CreateEpFactories returned ", num_factories, " factories but the supplied capacity was ", + new_factories.size(), "."); + + for (size_t i = 0; i < new_factories.size(); ++i) { + if ((i < num_factories) != (new_factories[i] != nullptr)) { + return ORT_MAKE_STATUS(ONNXRUNTIME, FAIL, + "CreateEpFactories returned factory outputs inconsistent with num_factories."); + } } + + factories.reserve(SafeInt(factories.size()) + num_factories); + factories.insert(factories.end(), + std::make_move_iterator(new_factories.begin()), + std::make_move_iterator(new_factories.begin() + num_factories)); + + return Status::OK(); } } // namespace ep_library_plugin_utils diff --git a/onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.h b/onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.h index 4f7255b5f8550..3b54f06663004 100644 --- a/onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.h +++ b/onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.h @@ -3,8 +3,8 @@ #pragma once +#include #include -#include #include #include "core/common/status.h" @@ -15,24 +15,24 @@ namespace ep_library_plugin_utils { // Shared implementation for EpLibrary types that obtain OrtEpFactory instances from a plugin EP's // CreateEpFactories and ReleaseEpFactory entry points. The entry points are resolved at load time for a -// dynamically loaded library and at link time for a statically linked one, but the calling convention and -// lifetime rules are the same either way. +// dynamically loaded library and at link time for a statically linked one, but the lifetime rules are the same either +// way. -/// -/// Calls the CreateEpFactories entry point and appends the factories it produces to `factories`. -/// `factories` is left unmodified if the entry point fails. -/// -Status CreateFactories(CreateEpApiFactoriesFn create_fn, const std::string& registration_name, - std::vector& factories); +struct OrtEpFactoryDeleter { + ReleaseEpApiFactoryFn release_fn{}; + + void operator()(OrtEpFactory* factory) const noexcept; +}; + +using OrtEpFactoryUniquePtr = std::unique_ptr; /// -/// Calls the ReleaseEpFactory entry point for every factory in `factories` and clears it. -/// Errors are logged rather than returned because the caller cannot meaningfully recover: the current -/// implementation assumes any failure is permanent and does not leave pieces around to re-attempt a release. -/// `library_description` identifies the library in log messages. +/// Calls the CreateEpFactories entry point and appends the factories it produces to `factories`. +/// `factories` is left unmodified and all returned factories are released if creation or validation fails. /// -void ReleaseFactories(ReleaseEpApiFactoryFn release_fn, std::vector& factories, - std::string_view library_description); +Status CreateFactories(CreateEpApiFactoriesFn create_fn, ReleaseEpApiFactoryFn release_fn, + const std::string& registration_name, + std::vector& factories); } // namespace ep_library_plugin_utils } // namespace onnxruntime diff --git a/onnxruntime/core/session/plugin_ep/ep_library_provider_bridge.cc b/onnxruntime/core/session/plugin_ep/ep_library_provider_bridge.cc index da94a9f12ba9d..ace8b273213e2 100644 --- a/onnxruntime/core/session/plugin_ep/ep_library_provider_bridge.cc +++ b/onnxruntime/core/session/plugin_ep/ep_library_provider_bridge.cc @@ -28,11 +28,10 @@ Status EpLibraryProviderBridge::Load() { // we also need to update any returned OrtEpDevice instances to swap the wrapper EpFactoryInternal in so that we can // call Provider::CreateIExecutionProvider in EpFactoryInternal::CreateIExecutionProvider. - for (const auto& factory : ep_library_plugin_->GetFactories()) { + for (OrtEpFactory* factory : ep_library_plugin_->GetFactories()) { auto factory_impl = std::make_unique(*factory, *provider_library_, library_path_); auto internal_factory = std::make_unique(std::move(factory_impl)); - factory_ptrs_.push_back(internal_factory.get()); internal_factory_ptrs_.push_back(internal_factory.get()); factories_.push_back(std::move(internal_factory)); } @@ -44,7 +43,6 @@ Status EpLibraryProviderBridge::Unload() { std::lock_guard lock{mutex_}; internal_factory_ptrs_.clear(); - factory_ptrs_.clear(); factories_.clear(); // we loaded ep_library_plugin_ after provider_library_ in LoadPluginOrProviderBridge so do the reverse order here. diff --git a/onnxruntime/core/session/plugin_ep/ep_library_provider_bridge.h b/onnxruntime/core/session/plugin_ep/ep_library_provider_bridge.h index f3147794bc823..46eecfd657cc7 100644 --- a/onnxruntime/core/session/plugin_ep/ep_library_provider_bridge.h +++ b/onnxruntime/core/session/plugin_ep/ep_library_provider_bridge.h @@ -36,10 +36,6 @@ class EpLibraryProviderBridge : public EpLibrary { return &library_path_; } - const std::vector& GetFactories() override { - return factory_ptrs_; - } - // Provider bridge EPs are 'internal' as they can provide an IExecutionProvider instance directly. const std::vector& GetInternalFactories() { return internal_factory_ptrs_; @@ -51,6 +47,14 @@ class EpLibraryProviderBridge : public EpLibrary { ORT_DISALLOW_COPY_AND_ASSIGNMENT(EpLibraryProviderBridge); private: + size_t GetFactoryCount() const override { + return factories_.size(); + } + + OrtEpFactory* GetFactory(size_t index) const override { + return factories_[index].get(); + } + std::mutex mutex_; std::unique_ptr provider_library_; // provider bridge EP library @@ -63,7 +67,6 @@ class EpLibraryProviderBridge : public EpLibrary { std::filesystem::path library_path_; std::vector> factories_; - std::vector factory_ptrs_; // for convenience std::vector internal_factory_ptrs_; // for convenience }; } // namespace onnxruntime diff --git a/onnxruntime/core/session/plugin_ep/ep_library_static_plugin.cc b/onnxruntime/core/session/plugin_ep/ep_library_static_plugin.cc index 95cae21e0d238..077d01abfe937 100644 --- a/onnxruntime/core/session/plugin_ep/ep_library_static_plugin.cc +++ b/onnxruntime/core/session/plugin_ep/ep_library_static_plugin.cc @@ -13,7 +13,7 @@ Status EpLibraryStaticPlugin::Load() { ORT_TRY { std::lock_guard lock{mutex_}; if (factories_.empty()) { - status = ep_library_plugin_utils::CreateFactories(create_fn_, registration_name_, factories_); + status = ep_library_plugin_utils::CreateFactories(create_fn_, release_fn_, registration_name_, factories_); } } ORT_CATCH(const std::exception& ex) { @@ -25,20 +25,13 @@ Status EpLibraryStaticPlugin::Load() { }); } - // Release anything that was created before the failure so a partial load does not leave the library - // half-initialized. There is no library handle to release. - if (!status.IsOK()) { - std::lock_guard lock{mutex_}; - ep_library_plugin_utils::ReleaseFactories(release_fn_, factories_, registration_name_); - } - return status; } Status EpLibraryStaticPlugin::Unload() { std::lock_guard lock{mutex_}; - ep_library_plugin_utils::ReleaseFactories(release_fn_, factories_, registration_name_); + factories_.clear(); return Status::OK(); } diff --git a/onnxruntime/core/session/plugin_ep/ep_library_static_plugin.h b/onnxruntime/core/session/plugin_ep/ep_library_static_plugin.h index 617b11baeca22..a974e8b88ccfd 100644 --- a/onnxruntime/core/session/plugin_ep/ep_library_static_plugin.h +++ b/onnxruntime/core/session/plugin_ep/ep_library_static_plugin.h @@ -8,6 +8,7 @@ #include #include "core/session/plugin_ep/ep_library.h" +#include "core/session/plugin_ep/ep_library_plugin_utils.h" namespace onnxruntime { /// @@ -37,19 +38,23 @@ class EpLibraryStaticPlugin : public EpLibrary { Status Load() override; - const std::vector& GetFactories() override { - return factories_; - } - Status Unload() override; ORT_DISALLOW_COPY_AND_ASSIGNMENT(EpLibraryStaticPlugin); private: + size_t GetFactoryCount() const override { + return factories_.size(); + } + + OrtEpFactory* GetFactory(size_t index) const override { + return factories_[index].get(); + } + std::mutex mutex_; const std::string registration_name_; const CreateEpApiFactoriesFn create_fn_; const ReleaseEpApiFactoryFn release_fn_; - std::vector factories_{}; + std::vector factories_{}; }; } // namespace onnxruntime diff --git a/onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc b/onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc index 7f1bed3de45d3..1650bb5cca542 100644 --- a/onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc +++ b/onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc @@ -25,6 +25,7 @@ #include "core/session/abi_logger.h" #include "core/session/abi_session_options_impl.h" #include "core/session/allocator_adapters.h" +#include "core/session/plugin_ep/ep_allocator_utils.h" #include "core/session/plugin_ep/ep_kernel_registration.h" #include "core/session/plugin_ep/ep_event_profiling.h" #include "core/session/ort_apis.h" @@ -861,16 +862,17 @@ std::unique_ptr PluginExecutionProvider::GetDataTran if (ep_factory_.CreateDataTransfer != nullptr) { OrtStatus* status = ep_factory_.CreateDataTransfer(&ep_factory_, &data_transfer_impl); + plugin_ep::OrtDataTransferImplPtr owned_impl{data_transfer_impl}; if (status != nullptr) { ORT_THROW("Error creating data transfer: ", ToStatusAndRelease(status).ToString()); } - } - if (data_transfer_impl == nullptr) { - return {}; + if (owned_impl != nullptr) { + return std::make_unique(std::move(owned_impl)); + } } - return std::make_unique(*data_transfer_impl); + return {}; } std::vector PluginExecutionProvider::CreatePreferredAllocators() { @@ -879,48 +881,19 @@ std::vector PluginExecutionProvider::CreatePreferredAllocators() { const OrtKeyValuePairs* allocator_options = session_arena_options_ ? &*session_arena_options_ : nullptr; - for (const auto* memory_info : allocator_mem_infos_) { - OrtAllocator* ort_allocator_ptr = nullptr; + auto* ep_factory = &ep_factory_; + for (const auto* memory_info : allocator_mem_infos_) { if (!ort_ep_->CreateAllocator && !ep_factory_.CreateAllocator) { ORT_THROW("The OrtEpDevice requires the EP library to implement an allocator, but none were found."); } - // prefer OrtEp function if available, otherwise fall back to using the OrtEpFactory implementation. - OrtStatus* ort_status = ort_ep_->CreateAllocator - ? ort_ep_->CreateAllocator(ort_ep_.get(), memory_info, &ort_allocator_ptr) - : ep_factory_.CreateAllocator(&ep_factory_, memory_info, allocator_options, - &ort_allocator_ptr); - - // throw or log? start with throw - if (ort_status != nullptr) { - ORT_THROW("Error creating allocator: ", ToStatusAndRelease(ort_status).ToString()); - } - - if (ort_allocator_ptr->Info(ort_allocator_ptr)->alloc_type == OrtAllocatorType::OrtArenaAllocator) { - ORT_THROW( - "OrtEpFactory returned an allocator with OrtAllocatorType of OrtArenaAllocator. " - "This type is reserved for ONNX Runtime internal usage only, as any arena usage by the " - "EP library should be opaque to ORT"); - } - - auto* ep_factory = &ep_factory_; - auto ort_allocator = OrtAllocatorUniquePtr( - ort_allocator_ptr, - [ep_factory](OrtAllocator* allocator) { - ep_factory->ReleaseAllocator(ep_factory, allocator); - }); - - // Use the arena wrapper when the allocator supports Shrink(), matching - // the logic in Environment::CreateSharedAllocatorImpl. This ensures - // per-session plugin arenas are visible to ShrinkMemoryArenas. AllocatorPtr alloc_ptr; - if (ort_allocator->version >= 25 && ort_allocator->Shrink != nullptr) { - alloc_ptr = std::make_shared(std::move(ort_allocator)); - } else { - alloc_ptr = std::make_shared(std::move(ort_allocator)); + ORT_THROW_IF_ERROR(ep_allocator_utils::CreateAndWrapEpAllocator(ort_ep_.get(), *ep_factory, *memory_info, + allocator_options, alloc_ptr)); + if (alloc_ptr != nullptr) { + allocators.push_back(std::move(alloc_ptr)); } - allocators.push_back(std::move(alloc_ptr)); } return allocators; diff --git a/onnxruntime/test/autoep/library/example_plugin_ep/ep_factory.cc b/onnxruntime/test/autoep/library/example_plugin_ep/ep_factory.cc index 7f3405d6ef0fd..fd64a848e52e7 100644 --- a/onnxruntime/test/autoep/library/example_plugin_ep/ep_factory.cc +++ b/onnxruntime/test/autoep/library/example_plugin_ep/ep_factory.cc @@ -11,6 +11,7 @@ #include "ep_arena.h" #include "ep_data_transfer.h" #include "ep_stream_support.h" +#include "ep_test_hooks.h" #include "core/session/onnxruntime_ep_device_ep_metadata_keys.h" #include "core/session/onnxruntime_session_options_config_keys.h" @@ -376,6 +377,10 @@ OrtStatus* ORT_API_CALL ExampleEpFactory::CreateDataTransferImpl(OrtEpFactory* t auto& factory = *static_cast(this_ptr); *data_transfer = factory.data_transfer_impl_.get(); + if (ShouldFailCreateDataTransfer()) { + return factory.ort_api.CreateStatus(ORT_FAIL, "injected data transfer creation failure"); + } + return nullptr; } diff --git a/onnxruntime/test/autoep/library/example_plugin_ep/ep_test_hooks.cc b/onnxruntime/test/autoep/library/example_plugin_ep/ep_test_hooks.cc index f09ed22869d10..675085f0c4dd6 100644 --- a/onnxruntime/test/autoep/library/example_plugin_ep/ep_test_hooks.cc +++ b/onnxruntime/test/autoep/library/example_plugin_ep/ep_test_hooks.cc @@ -8,6 +8,7 @@ std::atomic g_sync_count{0}; std::atomic g_preallocated_output_query_result{-1}; std::atomic g_preallocated_output_bad_index_rejected{-1}; +std::atomic g_create_data_transfer_should_fail{false}; extern "C" void ExampleEpTestHooks_ResetSyncCount() { g_sync_count.store(0); } extern "C" uint64_t ExampleEpTestHooks_GetSyncCount() { return g_sync_count.load(); } @@ -21,9 +22,15 @@ extern "C" int ExampleEpTestHooks_GetPreallocatedOutputQueryResult() { extern "C" int ExampleEpTestHooks_GetPreallocatedOutputBadIndexRejected() { return g_preallocated_output_bad_index_rejected.load(); } +extern "C" void ExampleEpTestHooks_SetCreateDataTransferFailure(int enabled) { + g_create_data_transfer_should_fail.store(enabled != 0); +} void RecordPreallocatedOutputQueryResult(int has_preallocated_output) { g_preallocated_output_query_result.store(has_preallocated_output); } void RecordPreallocatedOutputBadIndexRejected(int rejected) { g_preallocated_output_bad_index_rejected.store(rejected); } +bool ShouldFailCreateDataTransfer() { + return g_create_data_transfer_should_fail.load(); +} diff --git a/onnxruntime/test/autoep/library/example_plugin_ep/ep_test_hooks.h b/onnxruntime/test/autoep/library/example_plugin_ep/ep_test_hooks.h index 47168f0f5fce0..9a2ba88cde5cb 100644 --- a/onnxruntime/test/autoep/library/example_plugin_ep/ep_test_hooks.h +++ b/onnxruntime/test/autoep/library/example_plugin_ep/ep_test_hooks.h @@ -23,8 +23,10 @@ EXPORT_SYMBOL uint64_t ExampleEpTestHooks_GetSyncCount(); EXPORT_SYMBOL void ExampleEpTestHooks_ResetPreallocatedOutputQuery(); EXPORT_SYMBOL int ExampleEpTestHooks_GetPreallocatedOutputQueryResult(); EXPORT_SYMBOL int ExampleEpTestHooks_GetPreallocatedOutputBadIndexRejected(); +EXPORT_SYMBOL void ExampleEpTestHooks_SetCreateDataTransferFailure(int enabled); } // Internal to the library; not exported. void RecordPreallocatedOutputQueryResult(int has_preallocated_output); void RecordPreallocatedOutputBadIndexRejected(int rejected); +bool ShouldFailCreateDataTransfer(); diff --git a/onnxruntime/test/autoep/library/example_plugin_ep/example_plugin_ep_library.def b/onnxruntime/test/autoep/library/example_plugin_ep/example_plugin_ep_library.def index f3f87edbdb27b..59e909c86d84f 100644 --- a/onnxruntime/test/autoep/library/example_plugin_ep/example_plugin_ep_library.def +++ b/onnxruntime/test/autoep/library/example_plugin_ep/example_plugin_ep_library.def @@ -2,4 +2,3 @@ LIBRARY "example_plugin_ep.dll" EXPORTS CreateEpFactories @1 ReleaseEpFactory @2 - \ No newline at end of file diff --git a/onnxruntime/test/autoep/library/example_plugin_ep/example_plugin_ep_library.lds b/onnxruntime/test/autoep/library/example_plugin_ep/example_plugin_ep_library.lds index 5117e6f124b2d..bed5d9956be7b 100644 --- a/onnxruntime/test/autoep/library/example_plugin_ep/example_plugin_ep_library.lds +++ b/onnxruntime/test/autoep/library/example_plugin_ep/example_plugin_ep_library.lds @@ -7,6 +7,7 @@ VERS_1.0.0 { ExampleEpTestHooks_ResetPreallocatedOutputQuery; ExampleEpTestHooks_GetPreallocatedOutputQueryResult; ExampleEpTestHooks_GetPreallocatedOutputBadIndexRejected; + ExampleEpTestHooks_SetCreateDataTransferFailure; local: *; }; diff --git a/onnxruntime/test/autoep/test_autoep_utils.cc b/onnxruntime/test/autoep/test_autoep_utils.cc index 5e372b3dc58ee..d031f4504c349 100644 --- a/onnxruntime/test/autoep/test_autoep_utils.cc +++ b/onnxruntime/test/autoep/test_autoep_utils.cc @@ -92,6 +92,8 @@ void Utils::LoadExampleEpHooks(const ExamplePluginInfo& ep_info, hooks.get_preallocated_output_bad_index_rejected = reinterpret_cast( GetProcAddress(lib, "ExampleEpTestHooks_GetPreallocatedOutputBadIndexRejected")); + hooks.set_create_data_transfer_failure = reinterpret_cast( + GetProcAddress(lib, "ExampleEpTestHooks_SetCreateDataTransferFailure")); #else void* lib = dlopen(ep_info.library_path.c_str(), RTLD_LAZY | RTLD_LOCAL); ASSERT_NE(lib, nullptr); @@ -108,6 +110,8 @@ void Utils::LoadExampleEpHooks(const ExamplePluginInfo& ep_info, hooks.get_preallocated_output_bad_index_rejected = reinterpret_cast( dlsym(lib, "ExampleEpTestHooks_GetPreallocatedOutputBadIndexRejected")); + hooks.set_create_data_transfer_failure = reinterpret_cast( + dlsym(lib, "ExampleEpTestHooks_SetCreateDataTransferFailure")); #endif example_ep_hooks = LoadExampleEpHooksPtr( diff --git a/onnxruntime/test/autoep/test_autoep_utils.h b/onnxruntime/test/autoep/test_autoep_utils.h index 42f330a0124b9..cfdce5775ea45 100644 --- a/onnxruntime/test/autoep/test_autoep_utils.h +++ b/onnxruntime/test/autoep/test_autoep_utils.h @@ -40,12 +40,14 @@ struct Utils { using ResetPreallocatedOutputQueryFn = void (*)(); using GetPreallocatedOutputQueryResultFn = int (*)(); using GetPreallocatedOutputBadIndexRejectedFn = int (*)(); + using SetCreateDataTransferFailureFn = void (*)(int enabled); ResetSyncCountFn reset_sync_count{}; GetSyncCountFn get_sync_count{}; ResetPreallocatedOutputQueryFn reset_preallocated_output_query{}; GetPreallocatedOutputQueryResultFn get_preallocated_output_query_result{}; GetPreallocatedOutputBadIndexRejectedFn get_preallocated_output_bad_index_rejected{}; + SetCreateDataTransferFailureFn set_create_data_transfer_failure{}; }; using LoadExampleEpHooksPtr = std::unique_ptr>; diff --git a/onnxruntime/test/autoep/test_registration.cc b/onnxruntime/test/autoep/test_registration.cc index 4998f5e822854..20b00010cb64e 100644 --- a/onnxruntime/test/autoep/test_registration.cc +++ b/onnxruntime/test/autoep/test_registration.cc @@ -2,6 +2,7 @@ // Licensed under the MIT License. #include +#include #include #include @@ -107,6 +108,37 @@ TEST(OrtEpLibrary, LoadUnloadPluginLibraryCxxApi) { ort_env->UnregisterExecutionProviderLibrary(registration_name.c_str()); } +TEST(OrtEpLibrary, FailedRegistrationLeavesEnvironmentUnchanged) { + const std::filesystem::path& library_path = Utils::example_ep_info.library_path; + const std::string& registration_name = Utils::example_ep_info.registration_name; + const size_t initial_device_count = ort_env->GetEpDevices().size(); + + Utils::LoadExampleEpHooksPtr hooks; + ASSERT_NO_FATAL_FAILURE(Utils::LoadExampleEpHooks(Utils::example_ep_info, hooks)); + ASSERT_NE(hooks->set_create_data_transfer_failure, nullptr); + + { + // Fail registration with a data transfer creation failure. + hooks->set_create_data_transfer_failure(1); + auto reset_failure = gsl::finally([&] { hooks->set_create_data_transfer_failure(0); }); + + Ort::Status status{Ort::GetApi().RegisterExecutionProviderLibrary( + *ort_env, registration_name.c_str(), library_path.c_str())}; + ASSERT_FALSE(status.IsOK()); + + // The failed library must not contribute any EP devices to the environment. + EXPECT_EQ(ort_env->GetEpDevices().size(), initial_device_count); + } + + // The same registration name must remain available after the failed attempt. + ort_env->RegisterExecutionProviderLibrary(registration_name.c_str(), library_path.c_str()); + EXPECT_EQ(ort_env->GetEpDevices().size(), initial_device_count + 1); + ort_env->UnregisterExecutionProviderLibrary(registration_name.c_str()); + + // The successful registration must still support a normal unregister lifecycle. + EXPECT_EQ(ort_env->GetEpDevices().size(), initial_device_count); +} + // Test loading example_plugin_ep_virt_gpu and its associated OrtEpDevice/OrtHardwareDevice. // This EP creates a new OrtHardwareDevice instance that represents a virtual GPU and gives to ORT. TEST(OrtEpLibrary, LoadUnloadPluginVirtGpuLibraryCxxApi) { diff --git a/onnxruntime/test/framework/ep_library_plugin_utils_test.cc b/onnxruntime/test/framework/ep_library_plugin_utils_test.cc new file mode 100644 index 0000000000000..fa013d927c96a --- /dev/null +++ b/onnxruntime/test/framework/ep_library_plugin_utils_test.cc @@ -0,0 +1,110 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +#include "core/session/plugin_ep/ep_library_plugin_utils.h" + +#include "gtest/gtest.h" + +#include "core/session/onnxruntime_cxx_api.h" + +namespace onnxruntime::test { + +namespace { + +size_t released_factory_count = 0; + +OrtEpFactory* CreateTestFactory() { + return new OrtEpFactory{}; +} + +OrtStatus* ReleaseTestFactory(OrtEpFactory* factory) { + ++released_factory_count; + delete factory; + return nullptr; +} + +OrtStatus* CreateOneFactory(const char*, const OrtApiBase*, const OrtLogger*, + OrtEpFactory** factories, size_t, size_t* num_factories) { + factories[0] = CreateTestFactory(); + *num_factories = 1; + return nullptr; +} + +OrtStatus* CreateOneFactoryThenFail(const char*, const OrtApiBase*, const OrtLogger*, + OrtEpFactory** factories, size_t, size_t* num_factories) { + factories[0] = CreateTestFactory(); + *num_factories = 1; + return Ort::GetApi().CreateStatus(ORT_FAIL, "injected factory creation failure"); +} + +OrtStatus* ReturnOversizedFactoryCount(const char*, const OrtApiBase*, const OrtLogger*, + OrtEpFactory** factories, size_t max_factories, size_t* num_factories) { + factories[0] = CreateTestFactory(); + *num_factories = max_factories + 1; + return nullptr; +} + +OrtStatus* ReturnInconsistentFactoryOutputs(const char*, const OrtApiBase*, const OrtLogger*, + OrtEpFactory** factories, size_t, size_t* num_factories) { + factories[1] = CreateTestFactory(); + *num_factories = 1; + return nullptr; +} + +} // namespace + +class EpLibraryPluginUtilsTest : public ::testing::Test { + protected: + void SetUp() override { + released_factory_count = 0; + } +}; + +TEST_F(EpLibraryPluginUtilsTest, CreateFactoriesAdoptsFactoriesOnSuccess) { + std::vector factories; + + ASSERT_TRUE(ep_library_plugin_utils::CreateFactories( + CreateOneFactory, ReleaseTestFactory, "test plugin", factories) + .IsOK()); + ASSERT_EQ(factories.size(), 1u); + EXPECT_EQ(released_factory_count, 0u); + + factories.clear(); + EXPECT_TRUE(factories.empty()); + EXPECT_EQ(released_factory_count, 1u); +} + +TEST_F(EpLibraryPluginUtilsTest, CreateFactoriesReleasesFactoryWhenCreationFails) { + std::vector factories; + + const auto status = ep_library_plugin_utils::CreateFactories( + CreateOneFactoryThenFail, ReleaseTestFactory, "test plugin", factories); + + EXPECT_FALSE(status.IsOK()); + EXPECT_TRUE(factories.empty()); + EXPECT_EQ(released_factory_count, 1u); +} + +TEST_F(EpLibraryPluginUtilsTest, CreateFactoriesRejectsOversizedFactoryCount) { + std::vector factories; + + const auto status = ep_library_plugin_utils::CreateFactories( + ReturnOversizedFactoryCount, ReleaseTestFactory, "test plugin", factories); + + EXPECT_FALSE(status.IsOK()); + EXPECT_TRUE(factories.empty()); + EXPECT_EQ(released_factory_count, 1u); +} + +TEST_F(EpLibraryPluginUtilsTest, CreateFactoriesRejectsInconsistentFactoryOutputs) { + std::vector factories; + + const auto status = ep_library_plugin_utils::CreateFactories( + ReturnInconsistentFactoryOutputs, ReleaseTestFactory, "test plugin", factories); + + EXPECT_FALSE(status.IsOK()); + EXPECT_TRUE(factories.empty()); + EXPECT_EQ(released_factory_count, 1u); +} + +} // namespace onnxruntime::test diff --git a/onnxruntime/test/framework/ep_plugin_provider_test.cc b/onnxruntime/test/framework/ep_plugin_provider_test.cc index 4820f4e5c8898..5dedf2e88848b 100644 --- a/onnxruntime/test/framework/ep_plugin_provider_test.cc +++ b/onnxruntime/test/framework/ep_plugin_provider_test.cc @@ -18,6 +18,7 @@ #include "core/framework/config_options.h" #include "core/framework/kernel_def_builder.h" #include "core/framework/op_kernel.h" +#include "core/framework/plugin_data_transfer.h" #include "core/framework/resource_accountant.h" #include "core/graph/constants.h" #include "core/graph/graph_viewer.h" @@ -1405,6 +1406,7 @@ static FakeArenaOrtAllocator MakeFakeArenaAllocator(OrtMemoryInfo* mem_info, boo // Namespace-level storage so C function pointers can access the fake allocator. static OrtAllocator* g_fake_allocator_for_test = nullptr; +static int g_fake_allocator_release_count = 0; static OrtStatus* ORT_API_CALL FakeCreateAllocator(OrtEp*, const OrtMemoryInfo*, OrtAllocator** out) noexcept { @@ -1412,10 +1414,32 @@ static OrtStatus* ORT_API_CALL FakeCreateAllocator(OrtEp*, const OrtMemoryInfo*, return nullptr; } +static OrtStatus* ORT_API_CALL FakeCreateAllocatorWithError(OrtEp*, const OrtMemoryInfo*, + OrtAllocator** out) noexcept { + *out = g_fake_allocator_for_test; + return Ort::GetApi().CreateStatus(ORT_FAIL, "injected allocator creation failure"); +} + +static OrtStatus* ORT_API_CALL FakeCreateNullAllocator(OrtEp*, const OrtMemoryInfo*, + OrtAllocator** out) noexcept { + *out = nullptr; + return nullptr; +} + +static OrtStatus* ORT_API_CALL UnexpectedFactoryCreateAllocator(OrtEpFactory*, const OrtMemoryInfo*, + const OrtKeyValuePairs*, OrtAllocator** out) noexcept { + *out = nullptr; + return Ort::GetApi().CreateStatus(ORT_FAIL, "Unexpected call to OrtEpFactory::CreateAllocator"); +} + static void ORT_API_CALL FakeReleaseAllocator(OrtEpFactory*, OrtAllocator*) noexcept { // No-op: tests own the fake allocator lifetime. } +static void ORT_API_CALL FakeReleaseAllocatorCounting(OrtEpFactory*, OrtAllocator*) noexcept { + ++g_fake_allocator_release_count; +} + } // namespace TEST(PluginExecutionProviderTest, CreatePreferredAllocators_ShrinkCapableAllocatorExposedAsArena) { @@ -1436,6 +1460,7 @@ TEST(PluginExecutionProviderTest, CreatePreferredAllocators_ShrinkCapableAllocat g_fake_allocator_for_test = fake_alloc_ptr; ort_ep->CreateAllocator = FakeCreateAllocator; + test_plugin_ep::g_test_ort_ep_factory.CreateAllocator = UnexpectedFactoryCreateAllocator; test_plugin_ep::g_test_ort_ep_factory.ReleaseAllocator = FakeReleaseAllocator; auto allocators = ep->CreatePreferredAllocators(); @@ -1468,6 +1493,7 @@ TEST(PluginExecutionProviderTest, CreatePreferredAllocators_NonShrinkAllocatorNo g_fake_allocator_for_test = fake_alloc_ptr; ort_ep->CreateAllocator = FakeCreateAllocator; + test_plugin_ep::g_test_ort_ep_factory.CreateAllocator = UnexpectedFactoryCreateAllocator; test_plugin_ep::g_test_ort_ep_factory.ReleaseAllocator = FakeReleaseAllocator; auto allocators = ep->CreatePreferredAllocators(); @@ -1478,6 +1504,74 @@ TEST(PluginExecutionProviderTest, CreatePreferredAllocators_NonShrinkAllocatorNo << "Non-Shrink allocator must not be exposed as IArena"; } +TEST(PluginExecutionProviderTest, CreatePreferredAllocators_ReleasesReturnedAllocatorOnError) { + auto ort_device = test_plugin_ep::MakeTestOrtDevice(OrtDevice::GPU, OrtDevice::MemType::DEFAULT); + auto ort_memory_info = std::make_unique("FakeGPU", OrtAllocatorType::OrtDeviceAllocator, + ort_device, OrtMemTypeDefault); + auto fake_allocator = MakeFakeArenaAllocator(ort_memory_info.get()); + auto ort_hw_device = test_plugin_ep::MakeTestOrtHardwareDevice(OrtHardwareDeviceType_GPU); + auto ort_ep_device = test_plugin_ep::MakeTestOrtEpDevice(ort_hw_device.get(), ort_memory_info.get()); + std::vector ep_devices{ort_ep_device.get()}; + auto [ep, ort_ep] = test_plugin_ep::MakeTestOrtEp(ep_devices); + + g_fake_allocator_for_test = &fake_allocator; + g_fake_allocator_release_count = 0; + ort_ep->CreateAllocator = FakeCreateAllocatorWithError; + test_plugin_ep::g_test_ort_ep_factory.CreateAllocator = UnexpectedFactoryCreateAllocator; + test_plugin_ep::g_test_ort_ep_factory.ReleaseAllocator = FakeReleaseAllocatorCounting; + + try { + static_cast(ep->CreatePreferredAllocators()); + FAIL() << "Expected allocator creation to fail"; + } catch (const OnnxRuntimeException& ex) { + EXPECT_THAT(ex.what(), ::testing::HasSubstr("injected allocator creation failure")); + } + EXPECT_EQ(g_fake_allocator_release_count, 1); +} + +TEST(PluginExecutionProviderTest, CreatePreferredAllocators_AllowsNullAllocator) { + auto ort_device = test_plugin_ep::MakeTestOrtDevice(OrtDevice::GPU, OrtDevice::MemType::DEFAULT); + auto ort_memory_info = std::make_unique("FakeGPU", OrtAllocatorType::OrtDeviceAllocator, + ort_device, OrtMemTypeDefault); + auto ort_hw_device = test_plugin_ep::MakeTestOrtHardwareDevice(OrtHardwareDeviceType_GPU); + auto ort_ep_device = test_plugin_ep::MakeTestOrtEpDevice(ort_hw_device.get(), ort_memory_info.get()); + std::vector ep_devices{ort_ep_device.get()}; + auto [ep, ort_ep] = test_plugin_ep::MakeTestOrtEp(ep_devices); + + g_fake_allocator_release_count = 0; + ort_ep->CreateAllocator = FakeCreateNullAllocator; + test_plugin_ep::g_test_ort_ep_factory.CreateAllocator = UnexpectedFactoryCreateAllocator; + test_plugin_ep::g_test_ort_ep_factory.ReleaseAllocator = FakeReleaseAllocatorCounting; + + EXPECT_TRUE(ep->CreatePreferredAllocators().empty()); + EXPECT_EQ(g_fake_allocator_release_count, 0); +} + +TEST(PluginExecutionProviderTest, CreatePreferredAllocators_RejectsAllocatorWithNullMemoryInfo) { + auto ort_device = test_plugin_ep::MakeTestOrtDevice(OrtDevice::GPU, OrtDevice::MemType::DEFAULT); + auto ort_memory_info = std::make_unique("FakeGPU", OrtAllocatorType::OrtDeviceAllocator, + ort_device, OrtMemTypeDefault); + auto fake_allocator = MakeFakeArenaAllocator(nullptr); + auto ort_hw_device = test_plugin_ep::MakeTestOrtHardwareDevice(OrtHardwareDeviceType_GPU); + auto ort_ep_device = test_plugin_ep::MakeTestOrtEpDevice(ort_hw_device.get(), ort_memory_info.get()); + std::vector ep_devices{ort_ep_device.get()}; + auto [ep, ort_ep] = test_plugin_ep::MakeTestOrtEp(ep_devices); + + g_fake_allocator_for_test = &fake_allocator; + g_fake_allocator_release_count = 0; + ort_ep->CreateAllocator = FakeCreateAllocator; + test_plugin_ep::g_test_ort_ep_factory.CreateAllocator = UnexpectedFactoryCreateAllocator; + test_plugin_ep::g_test_ort_ep_factory.ReleaseAllocator = FakeReleaseAllocatorCounting; + + try { + static_cast(ep->CreatePreferredAllocators()); + FAIL() << "Expected allocator validation to fail"; + } catch (const OnnxRuntimeException& ex) { + EXPECT_THAT(ex.what(), ::testing::HasSubstr("OrtEp returned an allocator with null memory info.")); + } + EXPECT_EQ(g_fake_allocator_release_count, 1); +} + TEST(PluginExecutionProviderTest, IsGraphCaptureEnabled) { auto [ep, ort_ep] = test_plugin_ep::MakeTestOrtEp(); @@ -1862,4 +1956,66 @@ TEST(PluginExecutionProviderTest, OnSessionInitializationEnd_OldVersionFallback) ASSERT_STATUS_OK(ep->OnSessionInitializationEnd()); } +TEST(PluginDataTransferTest, OwnsAndReleasesImplementation) { + struct TestDataTransferImpl : OrtDataTransferImpl { + explicit TestDataTransferImpl(size_t& release_count) : release_count{release_count} { + Release = [](OrtDataTransferImpl* this_ptr) noexcept { + auto* impl = static_cast(this_ptr); + ++impl->release_count; + delete impl; + }; + CanCopy = [](const OrtDataTransferImpl*, const OrtMemoryDevice*, const OrtMemoryDevice*) noexcept { + return false; + }; + CopyTensors = [](OrtDataTransferImpl*, const OrtValue**, OrtValue**, OrtSyncStream**, size_t) noexcept { + return static_cast(nullptr); + }; + } + + size_t& release_count; + }; + + size_t release_count = 0; + { + plugin_ep::DataTransfer data_transfer{ + plugin_ep::OrtDataTransferImplPtr{new TestDataTransferImpl{release_count}}}; + } + + EXPECT_EQ(release_count, 1u); +} + +#if !defined(ORT_NO_EXCEPTIONS) +TEST(PluginDataTransferTest, RejectsImplementationWithMissingRequiredFunction) { + auto make_data_transfer_impl = []() { + OrtDataTransferImpl impl{}; + impl.Release = [](OrtDataTransferImpl*) noexcept {}; + impl.CanCopy = [](const OrtDataTransferImpl*, const OrtMemoryDevice*, const OrtMemoryDevice*) noexcept { + return false; + }; + impl.CopyTensors = [](OrtDataTransferImpl*, const OrtValue**, OrtValue**, OrtSyncStream**, size_t) noexcept { + return static_cast(nullptr); + }; + return impl; + }; + + { + auto impl = make_data_transfer_impl(); + impl.Release = nullptr; + EXPECT_THROW(plugin_ep::DataTransfer{plugin_ep::OrtDataTransferImplPtr{&impl}}, OnnxRuntimeException); + } + + { + auto impl = make_data_transfer_impl(); + impl.CanCopy = nullptr; + EXPECT_THROW(plugin_ep::DataTransfer{plugin_ep::OrtDataTransferImplPtr{&impl}}, OnnxRuntimeException); + } + + { + auto impl = make_data_transfer_impl(); + impl.CopyTensors = nullptr; + EXPECT_THROW(plugin_ep::DataTransfer{plugin_ep::OrtDataTransferImplPtr{&impl}}, OnnxRuntimeException); + } +} +#endif + } // namespace onnxruntime::test From ca3b31423d0a98ecf64c9f1d9cc1ba765bfbde94 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Tue, 22 Sep 2026 17:36:38 -0700 Subject: [PATCH 087/107] update comment --- onnxruntime/core/providers/webgpu/ep/api.cc | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/onnxruntime/core/providers/webgpu/ep/api.cc b/onnxruntime/core/providers/webgpu/ep/api.cc index 7f77c20f4f124..c0b1550d70cfd 100644 --- a/onnxruntime/core/providers/webgpu/ep/api.cc +++ b/onnxruntime/core/providers/webgpu/ep/api.cc @@ -12,10 +12,8 @@ #include "core/providers/webgpu/ep/factory.h" #include "core/session/onnxruntime_env_config_keys.h" -// When this EP is statically linked into the host binary instead of being built as a separate plugin library, the -// entry points below are given a `WebGpu_` prefix. ORT core declares the prefixed names in -// onnxruntime/core/session/plugin_ep/ep_static_plugins.cc, which spells them out literally, so the prefix is hardcoded -// on both sides rather than configured by the build. +// When this EP is statically linked into the host binary instead of being built as a separate shared library, the +// entry points below are given a `WebGpu_` prefix. // // The shared library build must leave the entry points unprefixed, because they are resolved by exact name. #if defined(ORT_PLUGIN_EP_STATICALLY_LINKED) From 1e53cbc29f82d614f359653abec2e2aa5ee47e78 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Tue, 22 Sep 2026 17:47:49 -0700 Subject: [PATCH 088/107] update --- onnxruntime/core/framework/plugin_data_transfer.h | 6 +++--- onnxruntime/core/session/environment.cc | 2 +- .../core/session/plugin_ep/ep_library_plugin_utils.h | 6 ++---- .../session/plugin_ep/ep_plugin_provider_interfaces.cc | 2 +- onnxruntime/test/framework/ep_plugin_provider_test.cc | 8 ++++---- 5 files changed, 11 insertions(+), 13 deletions(-) diff --git a/onnxruntime/core/framework/plugin_data_transfer.h b/onnxruntime/core/framework/plugin_data_transfer.h index f9be30cca2703..23ef3db98390c 100644 --- a/onnxruntime/core/framework/plugin_data_transfer.h +++ b/onnxruntime/core/framework/plugin_data_transfer.h @@ -22,7 +22,7 @@ struct OrtDataTransferImplDeleter { } }; -using OrtDataTransferImplPtr = std::unique_ptr; +using OrtDataTransferImplUniquePtr = std::unique_ptr; /// /// Class to implement IDataTransfer for plugin execution providers. @@ -30,7 +30,7 @@ using OrtDataTransferImplPtr = std::unique_ptr class DataTransfer : public IDataTransfer { public: - explicit DataTransfer(OrtDataTransferImplPtr impl) + explicit DataTransfer(OrtDataTransferImplUniquePtr impl) : impl_{std::move(impl)} { ORT_ENFORCE(impl_ != nullptr, "OrtDataTransferImpl must not be null."); ORT_ENFORCE(impl_->Release != nullptr, "OrtDataTransferImpl must provide a Release function."); @@ -58,7 +58,7 @@ class DataTransfer : public IDataTransfer { private: Status CopyTensorImpl(const Tensor& src, Tensor& dst, onnxruntime::Stream* stream = nullptr) const; - OrtDataTransferImplPtr impl_; + OrtDataTransferImplUniquePtr impl_; }; } // namespace plugin_ep } // namespace onnxruntime diff --git a/onnxruntime/core/session/environment.cc b/onnxruntime/core/session/environment.cc index cb9565dae7070..0680a2c182166 100644 --- a/onnxruntime/core/session/environment.cc +++ b/onnxruntime/core/session/environment.cc @@ -547,7 +547,7 @@ Status CreateDataTransferForFactory(OrtEpFactory& ep_factory, std::unique_ptr& data_transfer) { OrtDataTransferImpl* data_transfer_impl = nullptr; OrtStatus* ort_status = ep_factory.CreateDataTransfer(&ep_factory, &data_transfer_impl); - plugin_ep::OrtDataTransferImplPtr owned_impl{data_transfer_impl}; + plugin_ep::OrtDataTransferImplUniquePtr owned_impl{data_transfer_impl}; if (ort_status != nullptr) { return ORT_MAKE_STATUS(ONNXRUNTIME, EP_FAIL, diff --git a/onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.h b/onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.h index 3b54f06663004..01aecadcc7a8d 100644 --- a/onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.h +++ b/onnxruntime/core/session/plugin_ep/ep_library_plugin_utils.h @@ -26,10 +26,8 @@ struct OrtEpFactoryDeleter { using OrtEpFactoryUniquePtr = std::unique_ptr; -/// -/// Calls the CreateEpFactories entry point and appends the factories it produces to `factories`. -/// `factories` is left unmodified and all returned factories are released if creation or validation fails. -/// +// Calls the CreateEpFactories entry point and appends the factories it produces to `factories`. +// `factories` is left unmodified and all returned factories are released if creation or validation fails. Status CreateFactories(CreateEpApiFactoriesFn create_fn, ReleaseEpApiFactoryFn release_fn, const std::string& registration_name, std::vector& factories); diff --git a/onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc b/onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc index 1650bb5cca542..fe8bd44030b65 100644 --- a/onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc +++ b/onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc @@ -862,7 +862,7 @@ std::unique_ptr PluginExecutionProvider::GetDataTran if (ep_factory_.CreateDataTransfer != nullptr) { OrtStatus* status = ep_factory_.CreateDataTransfer(&ep_factory_, &data_transfer_impl); - plugin_ep::OrtDataTransferImplPtr owned_impl{data_transfer_impl}; + plugin_ep::OrtDataTransferImplUniquePtr owned_impl{data_transfer_impl}; if (status != nullptr) { ORT_THROW("Error creating data transfer: ", ToStatusAndRelease(status).ToString()); } diff --git a/onnxruntime/test/framework/ep_plugin_provider_test.cc b/onnxruntime/test/framework/ep_plugin_provider_test.cc index 5dedf2e88848b..eb744059a3e99 100644 --- a/onnxruntime/test/framework/ep_plugin_provider_test.cc +++ b/onnxruntime/test/framework/ep_plugin_provider_test.cc @@ -1978,7 +1978,7 @@ TEST(PluginDataTransferTest, OwnsAndReleasesImplementation) { size_t release_count = 0; { plugin_ep::DataTransfer data_transfer{ - plugin_ep::OrtDataTransferImplPtr{new TestDataTransferImpl{release_count}}}; + plugin_ep::OrtDataTransferImplUniquePtr{new TestDataTransferImpl{release_count}}}; } EXPECT_EQ(release_count, 1u); @@ -2001,19 +2001,19 @@ TEST(PluginDataTransferTest, RejectsImplementationWithMissingRequiredFunction) { { auto impl = make_data_transfer_impl(); impl.Release = nullptr; - EXPECT_THROW(plugin_ep::DataTransfer{plugin_ep::OrtDataTransferImplPtr{&impl}}, OnnxRuntimeException); + EXPECT_THROW(plugin_ep::DataTransfer{plugin_ep::OrtDataTransferImplUniquePtr{&impl}}, OnnxRuntimeException); } { auto impl = make_data_transfer_impl(); impl.CanCopy = nullptr; - EXPECT_THROW(plugin_ep::DataTransfer{plugin_ep::OrtDataTransferImplPtr{&impl}}, OnnxRuntimeException); + EXPECT_THROW(plugin_ep::DataTransfer{plugin_ep::OrtDataTransferImplUniquePtr{&impl}}, OnnxRuntimeException); } { auto impl = make_data_transfer_impl(); impl.CopyTensors = nullptr; - EXPECT_THROW(plugin_ep::DataTransfer{plugin_ep::OrtDataTransferImplPtr{&impl}}, OnnxRuntimeException); + EXPECT_THROW(plugin_ep::DataTransfer{plugin_ep::OrtDataTransferImplUniquePtr{&impl}}, OnnxRuntimeException); } } #endif From b4070c993b1b58183d8c57a630df31975f3130be Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Tue, 22 Sep 2026 18:19:07 -0700 Subject: [PATCH 089/107] fix compile issue --- onnxruntime/core/framework/plugin_data_transfer.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/onnxruntime/core/framework/plugin_data_transfer.cc b/onnxruntime/core/framework/plugin_data_transfer.cc index af57369ca50c5..9b80d5e24e7ec 100644 --- a/onnxruntime/core/framework/plugin_data_transfer.cc +++ b/onnxruntime/core/framework/plugin_data_transfer.cc @@ -59,7 +59,7 @@ Status DataTransfer::CopyTensorImpl(const Tensor& src_tensor, Tensor& dst_tensor const OrtValue* src_ptr = &src; OrtValue* dst_ptr = &dst; OrtSyncStream* stream_ptr = reinterpret_cast(stream); - auto* status = impl_->CopyTensors(&impl_, &src_ptr, &dst_ptr, &stream_ptr, 1); + auto* status = impl_->CopyTensors(impl_.get(), &src_ptr, &dst_ptr, &stream_ptr, 1); return ToStatusAndRelease(status); } From d72042e2e5582480c8565ccbc9b30c923ae4d619 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Tue, 22 Sep 2026 20:02:50 -0700 Subject: [PATCH 090/107] Fix plugin data transfer ownership test Update the data transfer manager test to transfer ownership of its plugin implementation through OrtDataTransferImplUniquePtr. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .../framework/data_transfer_manager_test.cc | 25 ++++++++++++------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/onnxruntime/test/framework/data_transfer_manager_test.cc b/onnxruntime/test/framework/data_transfer_manager_test.cc index 90da354cbc449..f50ba7c00656c 100644 --- a/onnxruntime/test/framework/data_transfer_manager_test.cc +++ b/onnxruntime/test/framework/data_transfer_manager_test.cc @@ -19,7 +19,12 @@ TEST(DataTransferManagerTest, PluginCopiesForwardStreams) { struct TestDataTransfer final : OrtDataTransferImpl { TestDataTransfer() : OrtDataTransferImpl{} { ort_version_supported = ORT_API_VERSION; - Release = [](OrtDataTransferImpl*) noexcept {}; + Release = [](OrtDataTransferImpl* impl) noexcept { + delete static_cast(impl); + }; + CanCopy = [](const OrtDataTransferImpl*, const OrtMemoryDevice*, const OrtMemoryDevice*) noexcept { + return true; + }; CopyTensors = [](OrtDataTransferImpl* impl, const OrtValue**, OrtValue**, OrtSyncStream** streams, size_t num_tensors) noexcept -> OrtStatus* { auto& self = *static_cast(impl); @@ -33,9 +38,11 @@ TEST(DataTransferManagerTest, PluginCopiesForwardStreams) { size_t copied_tensors = 0; OrtSyncStream* last_stream = nullptr; - } impl; + }; - plugin_ep::DataTransfer data_transfer{impl}; + auto impl = std::make_unique(); + auto* impl_ptr = impl.get(); + plugin_ep::DataTransfer data_transfer{plugin_ep::OrtDataTransferImplUniquePtr{impl.release()}}; auto allocator = TestCPUExecutionProvider()->CreatePreferredAllocators()[0]; Tensor source{DataTypeImpl::GetType(), TensorShape{4}, allocator}; Tensor destination{DataTypeImpl::GetType(), TensorShape{4}, allocator}; @@ -43,16 +50,16 @@ TEST(DataTransferManagerTest, PluginCopiesForwardStreams) { Stream stream{nullptr, device}; ASSERT_STATUS_OK(data_transfer.CopyTensorAsync(source, destination, stream)); - EXPECT_EQ(impl.copied_tensors, 1U); - EXPECT_EQ(impl.last_stream, reinterpret_cast(&stream)); + EXPECT_EQ(impl_ptr->copied_tensors, 1U); + EXPECT_EQ(impl_ptr->last_stream, reinterpret_cast(&stream)); ASSERT_STATUS_OK(data_transfer.CopyTensor(source, destination)); - EXPECT_EQ(impl.copied_tensors, 2U); - EXPECT_EQ(impl.last_stream, nullptr); + EXPECT_EQ(impl_ptr->copied_tensors, 2U); + EXPECT_EQ(impl_ptr->last_stream, nullptr); ASSERT_STATUS_OK(data_transfer.CopyTensors({{source, destination, &stream}})); - EXPECT_EQ(impl.copied_tensors, 3U); - EXPECT_EQ(impl.last_stream, reinterpret_cast(&stream)); + EXPECT_EQ(impl_ptr->copied_tensors, 3U); + EXPECT_EQ(impl_ptr->last_stream, reinterpret_cast(&stream)); } // DataTransferManager::CopyTensors should validate sizes match before calling the IDataTransfer implementation From f393cb8678eb3b92e797cfc56a12b27ddaa789e7 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Wed, 23 Sep 2026 10:18:58 -0700 Subject: [PATCH 091/107] fix build issue, update EpInfo::Create to set output ptr iff successful. --- include/onnxruntime/core/session/environment.h | 7 ++++--- onnxruntime/core/session/environment.cc | 6 +++--- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/include/onnxruntime/core/session/environment.h b/include/onnxruntime/core/session/environment.h index 6033c6e4edf13..503c3d2571b5c 100644 --- a/include/onnxruntime/core/session/environment.h +++ b/include/onnxruntime/core/session/environment.h @@ -286,9 +286,10 @@ class Environment { const std::vector& internal_factories = {}); struct EpInfo { - // calls EpLibrary::Load - // for each factory gets the OrtEpDevice instances and adds to execution_devices - // internal_factory is set if this is an internal EP + // Calls EpLibrary::Load. + // For each factory, gets the OrtEpDevice instances and adds to `out.execution_devices`. + // Provide `internal_factories` if this is an internal EP. + // If successful, `out` is set to the created instance. static Status Create(std::unique_ptr library_in, std::unique_ptr& out, const std::vector& internal_factories = {}); diff --git a/onnxruntime/core/session/environment.cc b/onnxruntime/core/session/environment.cc index d93bfbcfd4bde..aed7aecc9bd95 100644 --- a/onnxruntime/core/session/environment.cc +++ b/onnxruntime/core/session/environment.cc @@ -591,7 +591,6 @@ Status Environment::RegisterExecutionProviderLibrary(const std::string& registra std::unique_ptr ep_info; ORT_RETURN_IF_ERROR(EpInfo::Create(std::move(ep_library), ep_info, internal_factories)); - ORT_RETURN_IF_NOT(ep_info != nullptr); // Register shared allocators const size_t max_allocator_count = SafeInt(ep_info->execution_devices.size()) * 2; @@ -1023,8 +1022,8 @@ Status Environment::EpInfo::Create(std::unique_ptr library_in, std::u return ORT_MAKE_STATUS(ONNXRUNTIME, INVALID_ARGUMENT, "EpLibrary was null"); } - out.reset(new EpInfo()); // can't use make_unique with private ctor - EpInfo& instance = *out; + auto instance_unique_ptr = std::unique_ptr(new EpInfo()); // can't use make_unique with private ctor + EpInfo& instance = *instance_unique_ptr; instance.library = std::move(library_in); instance.internal_factories = internal_factories; @@ -1064,6 +1063,7 @@ Status Environment::EpInfo::Create(std::unique_ptr library_in, std::u } } + out = std::move(instance_unique_ptr); return Status::OK(); } From 07d5b4b6ff31c8dee08ad5fc45a024dd863d156f Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Mon, 28 Sep 2026 15:31:07 -0700 Subject: [PATCH 092/107] Fix plugin EP allocator callback validation Allow session-level allocator creation without requiring the factory creation callback, validate the shared release callback contract, and clarify the public API documentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../core/session/onnxruntime_ep_c_api.h | 19 +++- .../session/plugin_ep/ep_allocator_utils.cc | 20 +++-- .../session/plugin_ep/ep_allocator_utils.h | 7 +- .../test/framework/ep_plugin_provider_test.cc | 86 ++++++++++++++++++- 4 files changed, 117 insertions(+), 15 deletions(-) diff --git a/include/onnxruntime/core/session/onnxruntime_ep_c_api.h b/include/onnxruntime/core/session/onnxruntime_ep_c_api.h index 575d98a8fb8d3..fb9510de33846 100644 --- a/include/onnxruntime/core/session/onnxruntime_ep_c_api.h +++ b/include/onnxruntime/core/session/onnxruntime_ep_c_api.h @@ -2360,11 +2360,14 @@ struct OrtEp { * The OrtMemoryInfo instance will match one of the values set in the OrtEpDevice using EpDevice_AddAllocatorInfo. * Any allocator specific options should be read from the session options. * - * If nullptr OrtEpFactory::CreateAllocator will be used. + * Implementation of this function is optional. If nullptr, OrtEpFactory::CreateAllocator will be used if + * implemented. If neither callback is implemented, the EP does not provide a preferred allocator for the memory + * location. For default CPU memory, this allows ORT to use its default CPU allocator. * - * \param[in] this_ptr The OrtEpFactory instance. + * \param[in] this_ptr The OrtEp instance. * \param[in] memory_info The OrtMemoryInfo to create the allocator for. May be nullptr. - * \param[out] allocator The created OrtAllocator instance. Set to nullptr if the default CPU allocator is used. + * \param[out] allocator The created OrtAllocator instance. Set to nullptr if the EP does not provide a preferred + * allocator for the memory location. * * \snippet{doc} snippets.dox OrtStatus Return Value * @@ -2913,10 +2916,15 @@ struct OrtEpFactory { * The factory that creates the EP is responsible for providing the allocators required by the EP. * The OrtMemoryInfo instance will match one of the values set in the OrtEpDevice using EpDevice_AddAllocatorInfo. * + * Implementation of this function is optional. It is required if ORT needs to create a shared allocator for an + * OrtMemoryInfo advertised by the EP. + * * \param[in] this_ptr The OrtEpFactory instance. * \param[in] memory_info The OrtMemoryInfo to create the allocator for. May be nullptr. * \param[in] allocator_options Optional key-value pairs for allocator options, can be nullptr. - * \param[out] allocator The created OrtAllocator instance. Set to nullptr if the default CPU allocator is used. + * \param[out] allocator The created OrtAllocator instance. When called to create an allocator for a session, this + * may be set to nullptr if the EP does not provide a preferred allocator for the memory + * location. When called to create a shared allocator, this must be set to a non-null value. * * \snippet{doc} snippets.dox OrtStatus Return Value * @@ -2928,6 +2936,9 @@ struct OrtEpFactory { _Outptr_result_maybenull_ OrtAllocator** allocator); /** \brief Release an OrtAllocator created by the factory. + * + * Implementation of this function is required if OrtEpFactory::CreateAllocator or OrtEp::CreateAllocator is + * implemented. * * \since Version 1.23. */ diff --git a/onnxruntime/core/session/plugin_ep/ep_allocator_utils.cc b/onnxruntime/core/session/plugin_ep/ep_allocator_utils.cc index 2e501c21ee834..5772a71ded6c8 100644 --- a/onnxruntime/core/session/plugin_ep/ep_allocator_utils.cc +++ b/onnxruntime/core/session/plugin_ep/ep_allocator_utils.cc @@ -20,10 +20,20 @@ Status CreateAndWrapEpAllocator(OrtEp* ep, const OrtKeyValuePairs* ep_factory_allocator_options, AllocatorPtr& allocator_out, OrtAllocator** raw_allocator_out) { - ORT_RETURN_IF_NOT(ep_factory.CreateAllocator != nullptr && ep_factory.ReleaseAllocator != nullptr, - "OrtEpFactory must implement CreateAllocator and ReleaseAllocator."); + const bool has_ep_create = ep != nullptr && ep->CreateAllocator != nullptr; + const bool has_factory_create = ep_factory.CreateAllocator != nullptr; - const bool create_with_ep = ep != nullptr && ep->CreateAllocator != nullptr; + ORT_RETURN_IF_NOT((!has_ep_create && !has_factory_create) || ep_factory.ReleaseAllocator != nullptr, + "OrtEpFactory must implement ReleaseAllocator when OrtEp::CreateAllocator or " + "OrtEpFactory::CreateAllocator is implemented."); + + if (!has_ep_create && !has_factory_create) { + allocator_out = nullptr; + if (raw_allocator_out != nullptr) { + *raw_allocator_out = nullptr; + } + return Status::OK(); + } std::function allocator_deleter = [&ep_factory](OrtAllocator* allocator) { ep_factory.ReleaseAllocator(&ep_factory, allocator); @@ -31,7 +41,7 @@ Status CreateAndWrapEpAllocator(OrtEp* ep, OrtAllocator* raw_allocator = nullptr; OrtStatus* creation_ort_status = - create_with_ep + has_ep_create ? ep->CreateAllocator(ep, &memory_info, &raw_allocator) : ep_factory.CreateAllocator(&ep_factory, &memory_info, ep_factory_allocator_options, &raw_allocator); OrtAllocatorUniquePtr owned_allocator{raw_allocator, std::move(allocator_deleter)}; @@ -42,7 +52,7 @@ Status CreateAndWrapEpAllocator(OrtEp* ep, AllocatorPtr wrapped_allocator; if (owned_allocator != nullptr) { - const std::string_view creator = create_with_ep ? "OrtEp" : "OrtEpFactory"; + const std::string_view creator = has_ep_create ? "OrtEp" : "OrtEpFactory"; const OrtMemoryInfo* allocator_memory_info = owned_allocator->Info(owned_allocator.get()); ORT_RETURN_IF_NOT(allocator_memory_info != nullptr, creator, " returned an allocator with null memory info."); diff --git a/onnxruntime/core/session/plugin_ep/ep_allocator_utils.h b/onnxruntime/core/session/plugin_ep/ep_allocator_utils.h index 6785509d71608..81ea11ad3b571 100644 --- a/onnxruntime/core/session/plugin_ep/ep_allocator_utils.h +++ b/onnxruntime/core/session/plugin_ep/ep_allocator_utils.h @@ -10,8 +10,8 @@ namespace onnxruntime { namespace ep_allocator_utils { -// Creates a plugin EP allocator using OrtEp::CreateAllocator when available, otherwise OrtEpFactory::CreateAllocator, -// and wraps the result as an IAllocator. +// Creates a plugin EP allocator using OrtEp::CreateAllocator when available, otherwise OrtEpFactory::CreateAllocator +// when available, and wraps the result as an IAllocator. Returns a null allocator if neither callback is implemented. // Outputs are modified only on success. Status CreateAndWrapEpAllocator(OrtEp* ep, OrtEpFactory& ep_factory, @@ -20,7 +20,8 @@ Status CreateAndWrapEpAllocator(OrtEp* ep, AllocatorPtr& allocator_out, OrtAllocator** raw_allocator_out = nullptr); -// Creates a plugin EP allocator using OrtEpFactory::CreateAllocator and wraps the result as an IAllocator. +// Creates a plugin EP allocator using OrtEpFactory::CreateAllocator when available and wraps the result as an +// IAllocator. Returns a null allocator if the callback is not implemented. // Outputs are modified only on success. inline Status CreateAndWrapEpAllocator(OrtEpFactory& ep_factory, const OrtMemoryInfo& memory_info, diff --git a/onnxruntime/test/framework/ep_plugin_provider_test.cc b/onnxruntime/test/framework/ep_plugin_provider_test.cc index eb744059a3e99..8677ffe822712 100644 --- a/onnxruntime/test/framework/ep_plugin_provider_test.cc +++ b/onnxruntime/test/framework/ep_plugin_provider_test.cc @@ -27,6 +27,7 @@ #include "core/session/abi_devices.h" #include "core/session/onnxruntime_cxx_api.h" #include "core/session/onnxruntime_session_options_config_keys.h" +#include "core/session/plugin_ep/ep_allocator_utils.h" #include "test/util/include/api_asserts.h" #include "test/util/include/asserts.h" #include "test/util/include/test_environment.h" @@ -150,14 +151,15 @@ struct MakeTestOrtEpResult { // `setup` runs on the raw TestOrtEp before PluginExecutionProvider is constructed -- // callbacks consulted at construction time (e.g., GetDefaultMemoryDevice seeding // default_device_) must be configured here. -MakeTestOrtEpResult MakeTestOrtEp(std::vector ep_devices = {}, +MakeTestOrtEpResult MakeTestOrtEp(OrtEpFactory& ep_factory, + std::vector ep_devices, std::function setup = nullptr) { // Default OrtHardwareDevice and OrtEpDevice used if the caller does not explicitly provide ep_devices. static std::unique_ptr ort_hw_device = MakeTestOrtHardwareDevice(OrtHardwareDeviceType_CPU); static std::unique_ptr ort_ep_device = MakeTestOrtEpDevice(ort_hw_device.get()); auto ort_ep_raw = std::make_unique().release(); - auto ort_ep = UniqueOrtEp(ort_ep_raw, OrtEpDeleter{g_test_ort_ep_factory}); + auto ort_ep = UniqueOrtEp(ort_ep_raw, OrtEpDeleter{ep_factory}); if (setup) { setup(*ort_ep_raw); } @@ -170,7 +172,7 @@ MakeTestOrtEpResult MakeTestOrtEp(std::vector ep_devices = { auto& logging_manager = DefaultLoggingManager(); auto ep = std::make_unique(std::move(ort_ep), *static_cast(ort_session_options), - g_test_ort_ep_factory, + ep_factory, ep_devices, /*kernel_registry*/ nullptr, logging_manager.DefaultLogger()); @@ -179,6 +181,11 @@ MakeTestOrtEpResult MakeTestOrtEp(std::vector ep_devices = { return result; } +MakeTestOrtEpResult MakeTestOrtEp(std::vector ep_devices = {}, + std::function setup = nullptr) { + return MakeTestOrtEp(g_test_ort_ep_factory, std::move(ep_devices), std::move(setup)); +} + using LookUpKernelFunc = std::function; class MockKernelLookup : public IExecutionProvider::IKernelLookup { @@ -1504,6 +1511,7 @@ TEST(PluginExecutionProviderTest, CreatePreferredAllocators_NonShrinkAllocatorNo << "Non-Shrink allocator must not be exposed as IArena"; } +#if !defined(ORT_NO_EXCEPTIONS) TEST(PluginExecutionProviderTest, CreatePreferredAllocators_ReleasesReturnedAllocatorOnError) { auto ort_device = test_plugin_ep::MakeTestOrtDevice(OrtDevice::GPU, OrtDevice::MemType::DEFAULT); auto ort_memory_info = std::make_unique("FakeGPU", OrtAllocatorType::OrtDeviceAllocator, @@ -1528,6 +1536,7 @@ TEST(PluginExecutionProviderTest, CreatePreferredAllocators_ReleasesReturnedAllo } EXPECT_EQ(g_fake_allocator_release_count, 1); } +#endif // !defined(ORT_NO_EXCEPTIONS) TEST(PluginExecutionProviderTest, CreatePreferredAllocators_AllowsNullAllocator) { auto ort_device = test_plugin_ep::MakeTestOrtDevice(OrtDevice::GPU, OrtDevice::MemType::DEFAULT); @@ -1547,6 +1556,76 @@ TEST(PluginExecutionProviderTest, CreatePreferredAllocators_AllowsNullAllocator) EXPECT_EQ(g_fake_allocator_release_count, 0); } +TEST(PluginExecutionProviderTest, CreatePreferredAllocators_AllowsNullFactoryCreateAllocator) { + test_plugin_ep::TestOrtEpFactory ep_factory; + auto ort_device = test_plugin_ep::MakeTestOrtDevice(OrtDevice::GPU, OrtDevice::MemType::DEFAULT); + auto ort_memory_info = std::make_unique("FakeGPU", OrtAllocatorType::OrtDeviceAllocator, + ort_device, OrtMemTypeDefault); + auto fake_allocator = MakeFakeArenaAllocator(ort_memory_info.get()); + auto ort_hw_device = test_plugin_ep::MakeTestOrtHardwareDevice(OrtHardwareDeviceType_GPU); + auto ort_ep_device = + test_plugin_ep::MakeTestOrtEpDevice(ort_hw_device.get(), ep_factory, ort_memory_info.get(), nullptr); + std::vector ep_devices{ort_ep_device.get()}; + auto [ep, ort_ep] = test_plugin_ep::MakeTestOrtEp(ep_factory, ep_devices); + + g_fake_allocator_for_test = &fake_allocator; + ort_ep->CreateAllocator = FakeCreateAllocator; + ep_factory.CreateAllocator = nullptr; + ep_factory.ReleaseAllocator = FakeReleaseAllocator; + + auto allocators = ep->CreatePreferredAllocators(); + ASSERT_EQ(allocators.size(), 1u); +} + +TEST(PluginExecutionProviderTest, CreatePreferredAllocators_AllowsNoCreateAllocatorCallbacks) { + test_plugin_ep::TestOrtEpFactory ep_factory; + auto ort_device = test_plugin_ep::MakeTestOrtDevice(OrtDevice::GPU, OrtDevice::MemType::DEFAULT); + auto ort_memory_info = std::make_unique("FakeGPU", OrtAllocatorType::OrtDeviceAllocator, + ort_device, OrtMemTypeDefault); + auto ort_hw_device = test_plugin_ep::MakeTestOrtHardwareDevice(OrtHardwareDeviceType_GPU); + auto ort_ep_device = + test_plugin_ep::MakeTestOrtEpDevice(ort_hw_device.get(), ep_factory, ort_memory_info.get(), nullptr); + std::vector ep_devices{ort_ep_device.get()}; + auto [ep, ort_ep] = test_plugin_ep::MakeTestOrtEp(ep_factory, ep_devices); + + ort_ep->CreateAllocator = nullptr; + ep_factory.CreateAllocator = nullptr; + ep_factory.ReleaseAllocator = nullptr; + + EXPECT_TRUE(ep->CreatePreferredAllocators().empty()); +} + +TEST(PluginExecutionProviderTest, CreateAndWrapEpAllocator_AllowsNoCreateAllocatorCallbacks) { + test_plugin_ep::TestOrtEpFactory ep_factory; + auto ort_device = test_plugin_ep::MakeTestOrtDevice(OrtDevice::GPU, OrtDevice::MemType::DEFAULT); + OrtMemoryInfo ort_memory_info{"FakeGPU", OrtAllocatorType::OrtDeviceAllocator, ort_device, OrtMemTypeDefault}; + + AllocatorPtr allocator; + OrtAllocator* raw_allocator = nullptr; + ASSERT_STATUS_OK(ep_allocator_utils::CreateAndWrapEpAllocator( + ep_factory, ort_memory_info, nullptr, allocator, &raw_allocator)); + EXPECT_EQ(allocator, nullptr); + EXPECT_EQ(raw_allocator, nullptr); +} + +TEST(PluginExecutionProviderTest, CreateAndWrapEpAllocator_RejectsMissingReleaseAllocator) { + test_plugin_ep::TestOrtEpFactory ep_factory; + auto ort_device = test_plugin_ep::MakeTestOrtDevice(OrtDevice::GPU, OrtDevice::MemType::DEFAULT); + OrtMemoryInfo ort_memory_info{"FakeGPU", OrtAllocatorType::OrtDeviceAllocator, ort_device, OrtMemTypeDefault}; + test_plugin_ep::TestOrtEp ort_ep; + + ort_ep.CreateAllocator = FakeCreateAllocator; + ep_factory.CreateAllocator = nullptr; + ep_factory.ReleaseAllocator = nullptr; + + AllocatorPtr allocator; + const auto status = ep_allocator_utils::CreateAndWrapEpAllocator( + &ort_ep, ep_factory, ort_memory_info, nullptr, allocator); + ASSERT_FALSE(status.IsOK()); + EXPECT_THAT(status.ErrorMessage(), ::testing::HasSubstr("OrtEpFactory must implement ReleaseAllocator")); +} + +#if !defined(ORT_NO_EXCEPTIONS) TEST(PluginExecutionProviderTest, CreatePreferredAllocators_RejectsAllocatorWithNullMemoryInfo) { auto ort_device = test_plugin_ep::MakeTestOrtDevice(OrtDevice::GPU, OrtDevice::MemType::DEFAULT); auto ort_memory_info = std::make_unique("FakeGPU", OrtAllocatorType::OrtDeviceAllocator, @@ -1571,6 +1650,7 @@ TEST(PluginExecutionProviderTest, CreatePreferredAllocators_RejectsAllocatorWith } EXPECT_EQ(g_fake_allocator_release_count, 1); } +#endif // !defined(ORT_NO_EXCEPTIONS) TEST(PluginExecutionProviderTest, IsGraphCaptureEnabled) { auto [ep, ort_ep] = test_plugin_ep::MakeTestOrtEp(); From ece13109e8142c90c28675847009522448d9a547 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Mon, 28 Sep 2026 15:43:18 -0700 Subject: [PATCH 093/107] Trim GetSupportedDevices API remarks Remove implementation-specific locking and reentrancy guidance from the public plugin EP API documentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- include/onnxruntime/core/session/onnxruntime_ep_c_api.h | 4 ---- 1 file changed, 4 deletions(-) diff --git a/include/onnxruntime/core/session/onnxruntime_ep_c_api.h b/include/onnxruntime/core/session/onnxruntime_ep_c_api.h index fb9510de33846..7ebcc788f9f6a 100644 --- a/include/onnxruntime/core/session/onnxruntime_ep_c_api.h +++ b/include/onnxruntime/core/session/onnxruntime_ep_c_api.h @@ -2808,10 +2808,6 @@ struct OrtEpFactory { * Environment configuration entries are populated before any library is registered, so * OrtEpApi::GetEnvConfigEntries may be called from this function. * - * ORT holds internal environment locks across this call. An implementation must not call - * OrtApi::RegisterExecutionProviderLibrary or OrtApi::UnregisterExecutionProviderLibrary, as that - * deadlocks, and should otherwise limit the ORT APIs it calls and the work it does. - * * \since Version 1.22. */ ORT_API2_STATUS(GetSupportedDevices, _In_ OrtEpFactory* this_ptr, From fccebe906d7423cac3480128ead03d5822cc37e1 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Mon, 28 Sep 2026 16:04:41 -0700 Subject: [PATCH 094/107] Reuse status conversion helper in EP adapter Use ToStatusAndRelease for non-throwing kernel attribute access and remove the duplicate local conversion helper. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../onnxruntime/ep/adapter/op_kernel_info.h | 43 ++++++++----------- 1 file changed, 18 insertions(+), 25 deletions(-) diff --git a/include/onnxruntime/ep/adapter/op_kernel_info.h b/include/onnxruntime/ep/adapter/op_kernel_info.h index ebf3c22b189f7..49fb78ca6aaad 100644 --- a/include/onnxruntime/ep/adapter/op_kernel_info.h +++ b/include/onnxruntime/ep/adapter/op_kernel_info.h @@ -18,6 +18,7 @@ #include "core/common/narrow.h" #include "core/common/status.h" #include "core/framework/config_options.h" +#include "core/framework/error_code_helper.h" #include "core/framework/tensor_shape.h" #include "core/framework/tensor.h" @@ -204,36 +205,23 @@ struct OpKernelInfo { } private: - // A missing optional attribute is normal control flow for GetAttrOrDefault()/GetAttrsOrDefault(), so these - // accessors must not depend on catching the exception thrown by the Ort:: C++ wrappers: a plugin EP may be - // compiled with C++ exception catching disabled. The ORT Web build is one such case -- it builds with - // `-sDISABLE_EXCEPTION_CATCHING` everywhere except the C API boundary, so a `catch` in this inlined header - // never matches and the exception escapes all the way out of session creation. Use the non-throwing C API - // directly instead. - static Status ToStatus(OrtStatus* ort_status) { - if (ort_status == nullptr) { - return Status::OK(); - } - const Ort::Status status{ort_status}; // takes ownership - return Status(onnxruntime::common::ONNXRUNTIME, status.GetErrorCode(), status.GetErrorMessage()); - } - static Status GetAttrImpl(const OrtKernelInfo* info, const char* name, float* out) { - return ToStatus(Ort::GetApi().KernelInfoGetAttribute_float(info, name, out)); + return ToStatusAndRelease(Ort::GetApi().KernelInfoGetAttribute_float(info, name, out)); } static Status GetAttrImpl(const OrtKernelInfo* info, const char* name, int64_t* out) { - return ToStatus(Ort::GetApi().KernelInfoGetAttribute_int64(info, name, out)); + return ToStatusAndRelease(Ort::GetApi().KernelInfoGetAttribute_int64(info, name, out)); } static Status GetAttrImpl(const OrtKernelInfo* info, const char* name, std::string* out) { size_t size = 0; // Feed nullptr for the data buffer to query the true size of the string attribute. - ORT_RETURN_IF_ERROR(ToStatus(Ort::GetApi().KernelInfoGetAttribute_string(info, name, nullptr, &size))); + ORT_RETURN_IF_ERROR(ToStatusAndRelease(Ort::GetApi().KernelInfoGetAttribute_string(info, name, nullptr, &size))); std::string value; value.resize(size); - ORT_RETURN_IF_ERROR(ToStatus(Ort::GetApi().KernelInfoGetAttribute_string(info, name, value.data(), &size))); + ORT_RETURN_IF_ERROR( + ToStatusAndRelease(Ort::GetApi().KernelInfoGetAttribute_string(info, name, value.data(), &size))); value.resize(size - 1); // remove the terminating character '\0' *out = std::move(value); return Status::OK(); @@ -241,30 +229,34 @@ struct OpKernelInfo { static Status GetAttrsImpl(const OrtKernelInfo* info, const char* name, std::vector& out) { size_t size = 0; - ORT_RETURN_IF_ERROR(ToStatus(Ort::GetApi().KernelInfoGetAttributeArray_float(info, name, nullptr, &size))); + ORT_RETURN_IF_ERROR( + ToStatusAndRelease(Ort::GetApi().KernelInfoGetAttributeArray_float(info, name, nullptr, &size))); std::vector values(size); - ORT_RETURN_IF_ERROR(ToStatus(Ort::GetApi().KernelInfoGetAttributeArray_float(info, name, values.data(), &size))); + ORT_RETURN_IF_ERROR( + ToStatusAndRelease(Ort::GetApi().KernelInfoGetAttributeArray_float(info, name, values.data(), &size))); out.swap(values); return Status::OK(); } static Status GetAttrsImpl(const OrtKernelInfo* info, const char* name, std::vector& out) { size_t size = 0; - ORT_RETURN_IF_ERROR(ToStatus(Ort::GetApi().KernelInfoGetAttributeArray_int64(info, name, nullptr, &size))); + ORT_RETURN_IF_ERROR( + ToStatusAndRelease(Ort::GetApi().KernelInfoGetAttributeArray_int64(info, name, nullptr, &size))); std::vector values(size); - ORT_RETURN_IF_ERROR(ToStatus(Ort::GetApi().KernelInfoGetAttributeArray_int64(info, name, values.data(), &size))); + ORT_RETURN_IF_ERROR( + ToStatusAndRelease(Ort::GetApi().KernelInfoGetAttributeArray_int64(info, name, values.data(), &size))); out.swap(values); return Status::OK(); } static Status GetAttrsImpl(const OrtKernelInfo* info, const char* name, std::vector& out) { OrtAllocator* allocator = nullptr; - ORT_RETURN_IF_ERROR(ToStatus(Ort::GetApi().GetAllocatorWithDefaultOptions(&allocator))); + ORT_RETURN_IF_ERROR(ToStatusAndRelease(Ort::GetApi().GetAllocatorWithDefaultOptions(&allocator))); size_t size = 0; ORT_RETURN_IF_ERROR( - ToStatus(Ort::GetApi().KernelInfoGetAttributeArray_string(info, name, allocator, nullptr, &size))); + ToStatusAndRelease(Ort::GetApi().KernelInfoGetAttributeArray_string(info, name, allocator, nullptr, &size))); if (size == 0) { out.clear(); return Status::OK(); @@ -272,7 +264,8 @@ struct OpKernelInfo { char** raw_values = nullptr; ORT_RETURN_IF_ERROR( - ToStatus(Ort::GetApi().KernelInfoGetAttributeArray_string(info, name, allocator, &raw_values, &size))); + ToStatusAndRelease( + Ort::GetApi().KernelInfoGetAttributeArray_string(info, name, allocator, &raw_values, &size))); // The allocator owns both the array and every string in it, so release them on all exit paths: constructing the // std::string copies below can throw. Free through the OrtAllocator function pointer rather than From be525cc576019017aa0ddaf343f9e617f6b3b2ba Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Tue, 29 Sep 2026 09:07:08 -0700 Subject: [PATCH 095/107] Replace old use of routing_record with routing_snapshot_. --- onnxruntime/contrib_ops/cuda/moe/moe_quantization.cc | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/onnxruntime/contrib_ops/cuda/moe/moe_quantization.cc b/onnxruntime/contrib_ops/cuda/moe/moe_quantization.cc index 23f8c69fed824..6cc44c9e8eb05 100644 --- a/onnxruntime/contrib_ops/cuda/moe/moe_quantization.cc +++ b/onnxruntime/contrib_ops/cuda/moe/moe_quantization.cc @@ -1836,9 +1836,8 @@ Status QMoE::ComputeInternal(OpKernelContext* context) const { workspace_size, total_scratch_bytes); } #if !defined(BUILD_CUDA_EP_AS_PLUGIN) && !defined(ORT_MINIMAL_BUILD) - if (routing_record != nullptr) { - ORT_RETURN_IF_ERROR(routing_record->CaptureTile( - expert_indices, expert_scales, 0, SafeInt(num_rows) * SafeInt(k_), true, stream)); + if (routing_snapshot_) { + ORT_RETURN_IF_ERROR(routing_snapshot_->Consume()); } #endif return Status::OK(); From 503e433a9885a004db5616dfe9e7d562160a0ae6 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Tue, 29 Sep 2026 09:07:08 -0700 Subject: [PATCH 096/107] Replace old use of routing_record with routing_snapshot_. --- onnxruntime/contrib_ops/cuda/moe/moe_quantization.cc | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/onnxruntime/contrib_ops/cuda/moe/moe_quantization.cc b/onnxruntime/contrib_ops/cuda/moe/moe_quantization.cc index 23f8c69fed824..6cc44c9e8eb05 100644 --- a/onnxruntime/contrib_ops/cuda/moe/moe_quantization.cc +++ b/onnxruntime/contrib_ops/cuda/moe/moe_quantization.cc @@ -1836,9 +1836,8 @@ Status QMoE::ComputeInternal(OpKernelContext* context) const { workspace_size, total_scratch_bytes); } #if !defined(BUILD_CUDA_EP_AS_PLUGIN) && !defined(ORT_MINIMAL_BUILD) - if (routing_record != nullptr) { - ORT_RETURN_IF_ERROR(routing_record->CaptureTile( - expert_indices, expert_scales, 0, SafeInt(num_rows) * SafeInt(k_), true, stream)); + if (routing_snapshot_) { + ORT_RETURN_IF_ERROR(routing_snapshot_->Consume()); } #endif return Status::OK(); From 9d5c5763b5af31574d2abf755a9ff884903edb3a Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Tue, 29 Sep 2026 12:03:17 -0500 Subject: [PATCH 097/107] Fix packed QMoE expert snapshot capture Capture packed INT2 routing results before consuming the expert tracking snapshot and add a packed GEMV counting regression test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../contrib_ops/cuda/moe/moe_quantization.cc | 6 ++ .../framework/moe_expert_counting_test.cc | 85 ++++++++++++++----- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/onnxruntime/contrib_ops/cuda/moe/moe_quantization.cc b/onnxruntime/contrib_ops/cuda/moe/moe_quantization.cc index 6cc44c9e8eb05..c36167c6faacc 100644 --- a/onnxruntime/contrib_ops/cuda/moe/moe_quantization.cc +++ b/onnxruntime/contrib_ops/cuda/moe/moe_quantization.cc @@ -1772,6 +1772,12 @@ Status QMoE::ComputeInternal(OpKernelContext* context) const { const auto fused_routing = route_tile(0, num_rows); ORT_ENFORCE(fused_routing.router_logits == nullptr, "QMoE packed INT GEMV requires materialized routing outputs."); +#if !defined(BUILD_CUDA_EP_AS_PLUGIN) && !defined(ORT_MINIMAL_BUILD) + if (routing_snapshot_) { + ORT_RETURN_IF_ERROR(routing_snapshot_->Capture( + expert_indices, SafeInt(num_rows) * SafeInt(k_), stream)); + } +#endif const bool expert_maps_built = ck::fusedBuildExpertMapsSortFirstToken( expert_indices, p_r2u, unpermuted_row_to_permuted_row, p_exp, p_efto, num_rows, num_experts, static_cast(k_), 0, num_experts, stream); diff --git a/onnxruntime/test/framework/moe_expert_counting_test.cc b/onnxruntime/test/framework/moe_expert_counting_test.cc index 8563dc1223332..b38e2aec0a6f6 100644 --- a/onnxruntime/test/framework/moe_expert_counting_test.cc +++ b/onnxruntime/test/framework/moe_expert_counting_test.cc @@ -12,6 +12,7 @@ #include "core/graph/onnx_protobuf.h" #include "core/session/onnxruntime_session_options_config_keys.h" #include "gtest/gtest.h" +#include "test/common/cuda_op_test_utils.h" #include "test/test_environment.h" #include "test/unittest_util/framework_test_utils.h" #include "test/util/include/asserts.h" @@ -23,6 +24,8 @@ namespace onnxruntime::test { namespace { using namespace ONNX_NAMESPACE; constexpr int64_t kWidth = 128; +constexpr int64_t kPackedInt2GemvWidth = 512; +constexpr int64_t kPackedInt2GemvBlockSize = 64; constexpr int64_t kExperts = 4; void SetValue(ValueInfoProto& value, const std::string& name, int type, @@ -49,22 +52,31 @@ void AddZeroInitializer(GraphProto& graph, const char* name, int type, tensor->set_raw_data(std::string(size, '\0')); } -void PopulateMoeGraph(GraphProto& graph, bool quantized, bool cuda, bool subgraph, int64_t rows) { +void PopulateMoeGraph(GraphProto& graph, bool quantized, bool cuda, bool subgraph, int64_t rows, + bool use_packed_int2_gemv = false) { + const int64_t width = use_packed_int2_gemv ? kPackedInt2GemvWidth : kWidth; + const int64_t fc1_rows = use_packed_int2_gemv ? 2 * width : width; + const int64_t packed_width = quantized ? width / (use_packed_int2_gemv ? 4 : 2) : width; graph.set_name("expert_counting"); SetValue(subgraph ? *graph.add_value_info() : *graph.add_input(), "input", - TensorProto_DataType_FLOAT16, {rows, kWidth}); + TensorProto_DataType_FLOAT16, {rows, width}); SetValue(subgraph ? *graph.add_value_info() : *graph.add_input(), "router", TensorProto_DataType_FLOAT16, {rows, kExperts}); - SetValue(*graph.add_output(), "output", TensorProto_DataType_FLOAT16, {rows, kWidth}); + SetValue(*graph.add_output(), "output", TensorProto_DataType_FLOAT16, {rows, width}); const int weight_type = quantized ? TensorProto_DataType_UINT8 : TensorProto_DataType_FLOAT16; - AddZeroInitializer(graph, "w1", weight_type, {kExperts, kWidth, quantized ? kWidth / 2 : kWidth}, - quantized ? 1 : 2); - AddZeroInitializer(graph, "w2", weight_type, {kExperts, kWidth, quantized ? kWidth / 2 : kWidth}, - quantized ? 1 : 2); + AddZeroInitializer(graph, "w1", weight_type, {kExperts, fc1_rows, packed_width}, quantized ? 1 : 2); + AddZeroInitializer(graph, "w2", weight_type, {kExperts, width, packed_width}, quantized ? 1 : 2); if (quantized) { const int scale_type = cuda ? TensorProto_DataType_FLOAT16 : TensorProto_DataType_FLOAT; - AddZeroInitializer(graph, "s1", scale_type, {kExperts, kWidth}, cuda ? 2 : 4); - AddZeroInitializer(graph, "s2", scale_type, {kExperts, kWidth}, cuda ? 2 : 4); + if (use_packed_int2_gemv) { + AddZeroInitializer(graph, "s1", scale_type, {kExperts, fc1_rows, width / kPackedInt2GemvBlockSize}, + cuda ? 2 : 4); + AddZeroInitializer(graph, "s2", scale_type, {kExperts, width, width / kPackedInt2GemvBlockSize}, + cuda ? 2 : 4); + } else { + AddZeroInitializer(graph, "s1", scale_type, {kExperts, width}, cuda ? 2 : 4); + AddZeroInitializer(graph, "s2", scale_type, {kExperts, width}, cuda ? 2 : 4); + } } for (int i = 0; i < 2; ++i) { auto* node = graph.add_node(); @@ -86,11 +98,25 @@ void PopulateMoeGraph(GraphProto& graph, bool quantized, bool cuda, bool subgrap auto* activation = node->add_attribute(); activation->set_name("activation_type"); activation->set_type(AttributeProto_AttributeType_STRING); - activation->set_s("relu"); + activation->set_s(use_packed_int2_gemv ? "swiglu" : "relu"); + if (use_packed_int2_gemv) { + for (const auto& [name, value] : { + std::pair{"swiglu_fusion", int64_t{1}}, + std::pair{"expert_weight_bits", int64_t{2}}, + std::pair{"weights_prepacked", int64_t{0}}, + std::pair{"block_size", kPackedInt2GemvBlockSize}}) { + auto* attribute = node->add_attribute(); + attribute->set_name(name); + attribute->set_type(AttributeProto_AttributeType_INT); + attribute->set_i(value); + } + } } } -std::string MakeCountingModel(bool quantized = false, bool cuda = false, bool subgraphs = false, int64_t rows = 3) { +std::string MakeCountingModel(bool quantized = false, bool cuda = false, bool subgraphs = false, int64_t rows = 3, + bool use_packed_int2_gemv = false) { + ORT_ENFORCE(!use_packed_int2_gemv || (quantized && cuda && !subgraphs)); ModelProto model; model.set_ir_version(ONNX_NAMESPACE::Version::IR_VERSION); auto* opset = model.add_opset_import(); @@ -101,7 +127,7 @@ std::string MakeCountingModel(bool quantized = false, bool cuda = false, bool su opset->set_version(1); auto& graph = *model.mutable_graph(); if (!subgraphs) { - PopulateMoeGraph(graph, quantized, cuda, false, rows); + PopulateMoeGraph(graph, quantized, cuda, false, rows, use_packed_int2_gemv); } else { graph.set_name("conditional_counting"); SetValue(*graph.add_input(), "input", TensorProto_DataType_FLOAT16, {rows, kWidth}); @@ -122,12 +148,13 @@ std::string MakeCountingModel(bool quantized = false, bool cuda = false, bool su return model.SerializeAsString(); } -NameMLValMap CountingFeeds(bool subgraphs = false, bool condition = true, int64_t rows = 3) { +NameMLValMap CountingFeeds(bool subgraphs = false, bool condition = true, int64_t rows = 3, + int64_t width = kWidth) { ORT_ENFORCE(rows == 1 || rows == 3); auto allocator = TestCPUExecutionProvider()->CreatePreferredAllocators()[0]; OrtValue input, router, cond; - const std::vector values(rows * kWidth, MLFloat16(1.0f)); - CreateMLValue(allocator, {rows, kWidth}, values, &input); + const std::vector values(rows * width, MLFloat16(1.0f)); + CreateMLValue(allocator, {rows, width}, values, &input); const std::vector routing{ MLFloat16(9.f), MLFloat16(1.f), MLFloat16(0.f), MLFloat16(0.f), MLFloat16(8.f), MLFloat16(1.f), MLFloat16(0.f), MLFloat16(0.f), @@ -143,14 +170,16 @@ NameMLValMap CountingFeeds(bool subgraphs = false, bool condition = true, int64_ } Status ExecuteCountingModel(InferenceSession& session, std::vector& outputs, - bool subgraphs = false, bool condition = true, int64_t rows = 3) { + bool subgraphs = false, bool condition = true, int64_t rows = 3, + int64_t width = kWidth) { const std::array output_names{"output"}; - return session.Run(RunOptions{}, CountingFeeds(subgraphs, condition, rows), output_names, &outputs); + return session.Run(RunOptions{}, CountingFeeds(subgraphs, condition, rows, width), output_names, &outputs); } -void RunCountingModel(InferenceSession& session, bool subgraphs = false, bool condition = true, int64_t rows = 3) { +void RunCountingModel(InferenceSession& session, bool subgraphs = false, bool condition = true, int64_t rows = 3, + int64_t width = kWidth) { std::vector outputs; - ASSERT_STATUS_OK(ExecuteCountingModel(session, outputs, subgraphs, condition, rows)); + ASSERT_STATUS_OK(ExecuteCountingModel(session, outputs, subgraphs, condition, rows, width)); ASSERT_EQ(outputs.size(), 1U); for (auto value : outputs[0].Get().DataAsSpan()) { EXPECT_EQ(value.ToFloat(), 0.f); @@ -271,7 +300,8 @@ void TestDisabledRecording(bool quantized) { } } -void TestCounting(bool quantized, bool cuda, bool tiled = false, int64_t rows = 3) { +void TestCounting(bool quantized, bool cuda, bool tiled = false, int64_t rows = 3, + bool use_packed_int2_gemv = false) { auto options = CountingOptions(); if (tiled) { ASSERT_STATUS_OK(options.config_options.AddConfigEntry("ep.cuda.qmoe_row_tile_size", "1")); @@ -279,6 +309,10 @@ void TestCounting(bool quantized, bool cuda, bool tiled = false, int64_t rows = if (cuda) { ASSERT_STATUS_OK(options.config_options.AddConfigEntry(kOrtSessionOptionsDisableCPUEPFallback, "1")); } + if (use_packed_int2_gemv) { + // Make the dense INT2 fallback fail so this test cannot pass without selecting packed GEMV. + ASSERT_STATUS_OK(options.config_options.AddConfigEntry("ep.cuda.qmoe_int_dequant_max_scratch_bytes", "1")); + } InferenceSessionWrapper session(options, GetEnvironment()); if (cuda) { auto provider = DefaultCudaExecutionProvider(); @@ -290,7 +324,7 @@ void TestCounting(bool quantized, bool cuda, bool tiled = false, int64_t rows = } ASSERT_STATUS_OK(session.RegisterExecutionProvider(std::move(provider))); } - const auto model = MakeCountingModel(quantized, cuda, false, rows); + const auto model = MakeCountingModel(quantized, cuda, false, rows, use_packed_int2_gemv); ASSERT_STATUS_OK(session.Load(model.data(), static_cast(model.size()))); ASSERT_STATUS_OK(session.Initialize()); const auto* state = session.GetSessionState().GetMoeExpertState(); @@ -307,8 +341,9 @@ void TestCounting(bool quantized, bool cuda, bool tiled = false, int64_t rows = } } EXPECT_EQ(expert_ids.size(), 2U * kExperts); + const int64_t width = use_packed_int2_gemv ? kPackedInt2GemvWidth : kWidth; for (int run = 1; run <= 2; ++run) { - RunCountingModel(session, false, true, rows); + RunCountingModel(session, false, true, rows, width); for (size_t node_index : node_indices) { const double expected = run == 1 ? 0.1 : 0.19; InlinedVector counters; @@ -406,6 +441,12 @@ TEST(MoeExpertCountingTest, CudaMoE) { TestCounting(false, true); } TEST(MoeExpertCountingTest, CudaQMoE) { TestCounting(true, true); } TEST(MoeExpertCountingTest, CudaQMoETiled) { TestCounting(true, true, true); } TEST(MoeExpertCountingTest, CudaQMoESingleToken) { TestCounting(true, true, false, 1); } +TEST(MoeExpertCountingTest, CudaQMoEPackedInt2Gemv) { + if (!HasCudaEnvironment(800)) { + GTEST_SKIP() << "CUDA device with compute capability 8.0 or newer is required."; + } + TestCounting(true, true, false, 3, true); +} #endif TEST(MoeExpertCountingTest, DisabledAndIndependentSessions) { From 5ecef0fefac7f1f730af16352a1e6f35d09cf997 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Wed, 30 Sep 2026 10:45:49 -0700 Subject: [PATCH 098/107] simplify ifdefs --- onnxruntime/core/session/environment.cc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/onnxruntime/core/session/environment.cc b/onnxruntime/core/session/environment.cc index 2d321c2018b8b..fbadf6da73678 100644 --- a/onnxruntime/core/session/environment.cc +++ b/onnxruntime/core/session/environment.cc @@ -575,14 +575,14 @@ Status Environment::RegisterExecutionProviderLibrary(const std::string& registra const std::vector& internal_factories) { #if defined(_WIN32) || defined(ORT_USE_TELEMETRY) const Env& env = Env::Default(); -#endif + #if defined(ORT_USE_TELEMETRY) const TimePoint tp = std::chrono::high_resolution_clock::now(); #endif + // Windows ETW needs these calls even without ORT_USE_TELEMETRY. -#if defined(_WIN32) || defined(ORT_USE_TELEMETRY) env.GetTelemetryProvider().LogRegisterEpLibraryStart(registration_name); -#endif +#endif // defined(_WIN32) || defined(ORT_USE_TELEMETRY) auto status = Status::OK(); From 2cb6718e2a58f0ce749d2abe8b4a63a2a0ee8e89 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Wed, 30 Sep 2026 18:41:45 -0700 Subject: [PATCH 099/107] Handle WebGPU initialization edge cases Preserve legacy WebGPU selection when EP device APIs are unavailable in minimal builds, and document the synchronous GetEnvConfigEntries callback requirement. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../core/session/onnxruntime_ep_c_api.h | 4 +- js/web/lib/wasm/session-options.ts | 45 +++++++++++-------- 2 files changed, 29 insertions(+), 20 deletions(-) diff --git a/include/onnxruntime/core/session/onnxruntime_ep_c_api.h b/include/onnxruntime/core/session/onnxruntime_ep_c_api.h index 7ebcc788f9f6a..999cd495f354e 100644 --- a/include/onnxruntime/core/session/onnxruntime_ep_c_api.h +++ b/include/onnxruntime/core/session/onnxruntime_ep_c_api.h @@ -2806,7 +2806,9 @@ struct OrtEpFactory { * OrtApi::GetEpDevices since those depend on the execution provider library registration order. * * Environment configuration entries are populated before any library is registered, so - * OrtEpApi::GetEnvConfigEntries may be called from this function. + * OrtEpApi::GetEnvConfigEntries may be called from this function, but it must be called synchronously on + * the thread invoking GetSupportedDevices. The implementation must not wait for another thread that calls + * OrtEpApi::GetEnvConfigEntries. * * \since Version 1.22. */ diff --git a/js/web/lib/wasm/session-options.ts b/js/web/lib/wasm/session-options.ts index 424391fa59466..30f1b12185c58 100644 --- a/js/web/lib/wasm/session-options.ts +++ b/js/web/lib/wasm/session-options.ts @@ -70,20 +70,28 @@ const appendEpOption = (epOptions: Array<[number, number]>, key: string, value: epOptions.push([keyDataOffset, valueDataOffset]); }; +// OrtErrorCode values are part of the public C API ABI. +const ORT_NOT_IMPLEMENTED = 9; + /** * Get the OrtEpDevice instances registered with the environment, grouped by execution provider name. * * An execution provider may register more than one device, e.g. one per GPU, and all of them share its name. * - * @returns a map of execution provider name to the OrtEpDevice pointers registered under it. + * @returns a map of execution provider name to the OrtEpDevice pointers registered under it, or undefined if the + * build does not support the EP device API. */ -const getEpDevicesByEpName = (): Map => { +const getEpDevicesByEpName = (): Map | undefined => { const wasm = getInstance(); const stack = wasm.stackSave(); try { const epDevicesPtr = wasm.stackAlloc(wasm.PTR_SIZE); const numEpDevicesPtr = wasm.stackAlloc(wasm.PTR_SIZE); - if (wasm._OrtGetEpDevices(epDevicesPtr, numEpDevicesPtr) !== 0) { + const errorCode = wasm._OrtGetEpDevices(epDevicesPtr, numEpDevicesPtr); + if (errorCode === ORT_NOT_IMPLEMENTED) { + return undefined; + } + if (errorCode !== 0) { checkLastError("Can't get execution provider devices."); } // The array is owned by the environment, so only the pointers are read out here. @@ -117,8 +125,7 @@ const setExecutionProviders = async ( for (const ep of executionProviders) { let epName = typeof ep === 'string' ? ep : ep.name; const epOptions: Array<[number, number]> = []; - // True when the EP is selected by OrtEpDevice rather than by ORT's built-in EP name table. - let selectByEpDevice = false; + let selectedEpDevices: number[] | undefined; // check EP name switch (epName) { @@ -139,11 +146,18 @@ const setExecutionProviders = async ( break; case 'webgpu': if (!BUILD_DEFS.DISABLE_WEBGPU) { - // Select by OrtEpDevice, under the EP's canonical name (OrtEpFactory::GetName) rather than the short - // 'WebGPU' alias of the built-in EP name table. This is independent of how the paired wasm binary - // builds the WebGPU EP: the built-in EP and the plugin EP both register an OrtEpDevice under this name. - epName = 'WebGpuExecutionProvider'; - selectByEpDevice = true; + epName = 'WebGPU'; + const epDevicesByEpName = getEpDevicesByEpName(); + if (epDevicesByEpName !== undefined) { + // Non-minimal builds select by OrtEpDevice under the EP's canonical name. Minimal builds do not support + // the EP device API and continue to use the built-in EP name table. + epName = 'WebGpuExecutionProvider'; + selectedEpDevices = epDevicesByEpName.get(epName); + if (selectedEpDevices === undefined) { + const registered = [...epDevicesByEpName.keys()].join(', ') || '(none)'; + throw new Error(`no execution provider device is registered for: ${epName}. registered: ${registered}.`); + } + } let customDevice: GPUDevice | undefined; if (typeof ep !== 'string') { @@ -252,14 +266,7 @@ const setExecutionProviders = async ( } let appendErrorCode: number; - if (selectByEpDevice) { - const epDevicesByEpName = getEpDevicesByEpName(); - const epDevices = epDevicesByEpName.get(epName); - if (!epDevices) { - const registered = [...epDevicesByEpName.keys()].join(', ') || '(none)'; - throw new Error(`no execution provider device is registered for: ${epName}. registered: ${registered}.`); - } - + if (selectedEpDevices !== undefined) { // Select a single OrtEpDevice, even though more than one may match the EP name. // // Necessary: the WebGPU EP backs a session with one Dawn device, so its factory rejects any device @@ -275,7 +282,7 @@ const setExecutionProviders = async ( // "selected on a virtual GPU device" error rather than silently running on the wrong device. const epDevicesOffset = getInstance()._malloc(getInstance().PTR_SIZE); allocs.push(epDevicesOffset); - getInstance().setValue(epDevicesOffset, epDevices[0], '*'); + getInstance().setValue(epDevicesOffset, selectedEpDevices[0], '*'); appendErrorCode = await getInstance()._OrtAppendExecutionProviderV2( sessionOptionsHandle, From e064456dc7aa9ef5863186ef170cc8d45f2c29c7 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 1 Oct 2026 07:28:51 -0700 Subject: [PATCH 100/107] remove old yaml template reference --- .github/workflows/windows_webgpu.yml | 5 ----- 1 file changed, 5 deletions(-) diff --git a/.github/workflows/windows_webgpu.yml b/.github/workflows/windows_webgpu.yml index 4ca7f6334c484..d8c525ae8b557 100644 --- a/.github/workflows/windows_webgpu.yml +++ b/.github/workflows/windows_webgpu.yml @@ -476,11 +476,6 @@ jobs: with: architecture: x64 - - name: Install Visual Studio Spectre-mitigated libs - uses: ./.github/actions/install-vs-spectre-libs - with: - architecture: x64 - - name: Install python modules run: python -m pip install -r tools\ci_build\github\windows\python\requirements.txt shell: cmd From 665e9781815555cc63a765e84d7b992e9c40c46e Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 1 Oct 2026 07:34:16 -0700 Subject: [PATCH 101/107] increase Android minimal baseline build binary size threshold to 1601536 --- .github/workflows/android.yml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/.github/workflows/android.yml b/.github/workflows/android.yml index 5142f466bfbde..a198cfb7d53a5 100644 --- a/.github/workflows/android.yml +++ b/.github/workflows/android.yml @@ -79,8 +79,9 @@ jobs: run: | set -e -x BINARY_SIZE_THRESHOLD_ARGS="" - echo "Binary size threshold in bytes: 1599488" - BINARY_SIZE_THRESHOLD_ARGS="--threshold_size_in_bytes 1599488" + BINARY_SIZE_THRESHOLD_IN_BYTES=1601536 + echo "Binary size threshold in bytes: ${BINARY_SIZE_THRESHOLD_IN_BYTES}" + BINARY_SIZE_THRESHOLD_ARGS="--threshold_size_in_bytes ${BINARY_SIZE_THRESHOLD_IN_BYTES}" # Ensure ANDROID_NDK_HOME is available and get its real path if [ -z "$ANDROID_NDK_HOME" ]; then From 6b02a61648cd72a6c6d7cfa20ec18b170025617a Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 1 Oct 2026 19:33:55 -0700 Subject: [PATCH 102/107] Ignore virtual devices in WebGPU allocator tests Virtual WebGPU devices are compile-only and intentionally omit allocator information. Filter them explicitly using the standard hardware metadata key. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../webgpu/plugin/test_webgpu_allocators.cc | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/onnxruntime/test/providers/webgpu/plugin/test_webgpu_allocators.cc b/onnxruntime/test/providers/webgpu/plugin/test_webgpu_allocators.cc index c460bc39a5887..f1638dace927d 100644 --- a/onnxruntime/test/providers/webgpu/plugin/test_webgpu_allocators.cc +++ b/onnxruntime/test/providers/webgpu/plugin/test_webgpu_allocators.cc @@ -19,6 +19,7 @@ #include "core/graph/onnx_protobuf.h" #include "core/platform/env_var_utils.h" #include "core/session/onnxruntime_cxx_api.h" +#include "core/session/onnxruntime_ep_device_ep_metadata_keys.h" #include "core/session/onnxruntime_session_options_config_keys.h" #include "test/providers/webgpu/plugin/webgpu_plugin_test_utils.h" @@ -145,8 +146,17 @@ class WebGpuPluginSharedAllocatorTest : public ::testing::Test { protected: void SetUp() override { registration_.emplace(*ort_env, "webgpu_allocator_test"); + + const auto is_virtual_device = [](const Ort::ConstEpDevice& device) { + const auto metadata = device.Device().Metadata().GetKeyValuePairs(); + const auto is_virtual = metadata.find(kOrtHardwareDevice_MetadataKey_IsVirtual); + return is_virtual != metadata.end() && is_virtual->second == "1"; + }; + for (const auto& device : registration_->GetEpDevices()) { - ep_devices_.push_back(device); + if (!is_virtual_device(device)) { + ep_devices_.push_back(device); + } } ASSERT_FALSE(ep_devices_.empty()); } From 3bd332d3e56ec783449dfff41911d05625238769 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Thu, 1 Oct 2026 20:34:15 -0700 Subject: [PATCH 103/107] Fix WebGPU test provider option ambiguity Construct KeyValuePairs explicitly so GCC can select the intended AppendExecutionProvider_V2 overload. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../test/providers/webgpu/plugin/test_webgpu_concurrency.cc | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/onnxruntime/test/providers/webgpu/plugin/test_webgpu_concurrency.cc b/onnxruntime/test/providers/webgpu/plugin/test_webgpu_concurrency.cc index 9389919e8b3ef..bec01bf6a43c3 100644 --- a/onnxruntime/test/providers/webgpu/plugin/test_webgpu_concurrency.cc +++ b/onnxruntime/test/providers/webgpu/plugin/test_webgpu_concurrency.cc @@ -415,7 +415,9 @@ class PluginEpWebGpuConcurrency : public ::testing::Test { options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_DISABLE_ALL); options.AddConfigEntry(kOrtSessionOptionsDisableCPUEPFallback, "1"); options.EnableProfiling(ORT_TSTR("webgpu_copy_regression")); - options.AppendExecutionProvider_V2(*ort_env, {Device()}, {{"maxNumPendingDispatches", "4096"}}); + Ort::KeyValuePairs ep_options; + ep_options.Add("maxNumPendingDispatches", "4096"); + options.AppendExecutionProvider_V2(*ort_env, {Device()}, ep_options); const auto bytes = model.SerializeAsString(); Ort::Session session(*ort_env, bytes.data(), bytes.size(), options); for (const auto& device : session.GetEpDeviceForOutputs()) { From 74223309229b31431406e04417029ee91f9bf0d2 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Fri, 2 Oct 2026 06:40:33 -0700 Subject: [PATCH 104/107] Fix Linux WebGPU static plugin test build Suppress GCC strict-aliasing diagnostics from generated Dawn headers and use the existing Ort status helper in concurrency tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../webgpu/plugin/test_webgpu_allocators.cc | 7 ++++ .../webgpu/plugin/test_webgpu_concurrency.cc | 35 ++++++++----------- 2 files changed, 21 insertions(+), 21 deletions(-) diff --git a/onnxruntime/test/providers/webgpu/plugin/test_webgpu_allocators.cc b/onnxruntime/test/providers/webgpu/plugin/test_webgpu_allocators.cc index f1638dace927d..35dbd0d7a7025 100644 --- a/onnxruntime/test/providers/webgpu/plugin/test_webgpu_allocators.cc +++ b/onnxruntime/test/providers/webgpu/plugin/test_webgpu_allocators.cc @@ -26,8 +26,15 @@ #include "test/util/include/api_asserts.h" #if defined(ORT_UNIT_TEST_HAS_WEBGPU_PLUGIN_EP) && !defined(USE_EXTERNAL_DAWN) && !defined(BUILD_DAWN_SHARED_LIBRARY) +#if defined(__GNUC__) +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wstrict-aliasing" +#endif #include "dawn/dawn_proc.h" #include "dawn/native/DawnNative.h" +#if defined(__GNUC__) +#pragma GCC diagnostic pop +#endif #endif extern std::unique_ptr ort_env; diff --git a/onnxruntime/test/providers/webgpu/plugin/test_webgpu_concurrency.cc b/onnxruntime/test/providers/webgpu/plugin/test_webgpu_concurrency.cc index bec01bf6a43c3..c912980c4aa47 100644 --- a/onnxruntime/test/providers/webgpu/plugin/test_webgpu_concurrency.cc +++ b/onnxruntime/test/providers/webgpu/plugin/test_webgpu_concurrency.cc @@ -56,13 +56,6 @@ class FirstError { std::string message_; }; -void ThrowOnError(OrtStatus* status_ptr) { - Ort::Status status{status_ptr}; - if (!status.IsOK()) { - throw std::runtime_error(status.GetErrorMessage()); - } -} - constexpr int kThreads = 4; constexpr int kIterations = 20; constexpr size_t kElements = 6; @@ -112,8 +105,8 @@ void CopyTensorRoundTrip(Allocator& allocator, float value) { auto cpu_output = Ort::Value::CreateTensor( cpu_memory_info, output_data.data(), output_data.size(), kShape.data(), kShape.size()); - ThrowOnError(ort_env->CopyTensor(cpu_input, gpu_tensor, nullptr)); - ThrowOnError(ort_env->CopyTensor(gpu_tensor, cpu_output, nullptr)); + Ort::ThrowOnError(ort_env->CopyTensor(cpu_input, gpu_tensor, nullptr)); + Ort::ThrowOnError(ort_env->CopyTensor(gpu_tensor, cpu_output, nullptr)); if (output_data != input_data) { throw std::runtime_error("CopyTensor round trip returned incorrect data"); } @@ -162,7 +155,7 @@ class PluginEpWebGpuConcurrency : public ::testing::Test { Ort::Allocator allocator(session, Device().GetMemoryInfo(OrtDeviceMemoryType_DEFAULT)); auto gpu_input = Ort::Value::CreateTensor(allocator, kShape.data(), kShape.size()); auto gpu_output = Ort::Value::CreateTensor(allocator, kShape.data(), kShape.size()); - ThrowOnError(ort_env->CopyTensor(cpu_input, gpu_input, nullptr)); + Ort::ThrowOnError(ort_env->CopyTensor(cpu_input, gpu_input, nullptr)); Ort::IoBinding io_binding(session); io_binding.BindInput("X", gpu_input); @@ -171,7 +164,7 @@ class PluginEpWebGpuConcurrency : public ::testing::Test { session.Run(Ort::RunOptions{nullptr}, io_binding); io_binding.SynchronizeOutputs(); - ThrowOnError(ort_env->CopyTensor(gpu_output, cpu_output, nullptr)); + Ort::ThrowOnError(ort_env->CopyTensor(gpu_output, cpu_output, nullptr)); VerifyOutput(output_data, value); } @@ -208,7 +201,7 @@ class PluginEpWebGpuConcurrency : public ::testing::Test { std::array output_data{}; auto cpu_output = Ort::Value::CreateTensor( cpu_memory_info, output_data.data(), output_data.size(), kShape.data(), kShape.size()); - ThrowOnError(ort_env->CopyTensor(gpu_output, cpu_output, nullptr)); + Ort::ThrowOnError(ort_env->CopyTensor(gpu_output, cpu_output, nullptr)); VerifyOutput(output_data, value); } @@ -220,7 +213,7 @@ class PluginEpWebGpuConcurrency : public ::testing::Test { cpu_memory_info, input_data.data(), input_data.size(), kShape.data(), kShape.size()); Ort::Allocator allocator(session, Device().GetMemoryInfo(OrtDeviceMemoryType_DEFAULT)); auto gpu_input = Ort::Value::CreateTensor(allocator, kShape.data(), kShape.size()); - ThrowOnError(ort_env->CopyTensor(cpu_input, gpu_input, nullptr)); + Ort::ThrowOnError(ort_env->CopyTensor(cpu_input, gpu_input, nullptr)); const std::array input_names{"X"}; const std::array output_names{"Y"}; @@ -317,7 +310,7 @@ class PluginEpWebGpuConcurrency : public ::testing::Test { cpu_node_data[index] = static_cast(100 + iteration * kElements + index); } if (use_gpu_feed) { - ThrowOnError(ort_env->CopyTensor(cpu_node_source, feeds[reverse_feeds ? 0 : 1], nullptr)); + Ort::ThrowOnError(ort_env->CopyTensor(cpu_node_source, feeds[reverse_feeds ? 0 : 1], nullptr)); } if (bind_cpu_output_to_gpu) { for (size_t index = 0; index < feeds.size(); ++index) { @@ -337,7 +330,7 @@ class PluginEpWebGpuConcurrency : public ::testing::Test { Ort::Value::CreateTensor(allocator, kShape.data(), kShape.size()), Ort::Value::CreateTensor(allocator, kShape.data(), kShape.size())}; for (auto& cached_tensor : cached_tensors) { - ThrowOnError(ort_env->CopyTensor(feeds[reverse_feeds ? 1 : 0], cached_tensor, nullptr)); + Ort::ThrowOnError(ort_env->CopyTensor(feeds[reverse_feeds ? 1 : 0], cached_tensor, nullptr)); } } @@ -353,7 +346,7 @@ class PluginEpWebGpuConcurrency : public ::testing::Test { ASSERT_EQ(outputs[1].GetTensorMemoryInfo().GetDeviceType(), OrtMemoryInfoDeviceType_GPU); auto cpu_output = Ort::Value::CreateTensor( cpu_memory, cpu_output_data.data(), cpu_output_data.size(), kShape.data(), kShape.size()); - ThrowOnError(ort_env->CopyTensor(outputs[1], cpu_output, nullptr)); + Ort::ThrowOnError(ort_env->CopyTensor(outputs[1], cpu_output, nullptr)); outputs[1] = std::move(cpu_output); binding.ClearBoundOutputs(); } else { @@ -434,7 +427,7 @@ class PluginEpWebGpuConcurrency : public ::testing::Test { auto gpu_producer = Ort::Value::CreateTensor(allocator, shape.data(), shape.size()); auto gpu_copy = Ort::Value::CreateTensor(allocator, shape.data(), shape.size()); ASSERT_NE(gpu_producer.GetTensorMutableData(), gpu_copy.GetTensorMutableData()); - ThrowOnError(ort_env->CopyTensor(input, gpu_input, nullptr)); + Ort::ThrowOnError(ort_env->CopyTensor(input, gpu_input, nullptr)); Ort::IoBinding binding(session); binding.BindInput("X", gpu_input); // Preallocated, distinct graph outputs force the no-op kernel's actual copy branch, @@ -449,7 +442,7 @@ class PluginEpWebGpuConcurrency : public ::testing::Test { std::array output_data{}; auto output = Ort::Value::CreateTensor(cpu_memory, output_data.data(), output_data.size(), shape.data(), shape.size()); - ThrowOnError(ort_env->CopyTensor(gpu_copy, output, nullptr)); + Ort::ThrowOnError(ort_env->CopyTensor(gpu_copy, output, nullptr)); for (size_t i = 0; i < input_data.size(); ++i) { EXPECT_EQ(output_data[i], -input_data[i]); } @@ -758,7 +751,7 @@ TEST_F(PluginEpWebGpuConcurrency, DedicatedSessionAllocatorFeedsConcurrentSessio auto input = Ort::Value::CreateTensor( cpu_memory, input_data.data(), input_data.size(), kShape.data(), kShape.size()); auto gpu_input = Ort::Value::CreateTensor(allocator, kShape.data(), kShape.size()); - ThrowOnError(ort_env->CopyTensor(input, gpu_input, nullptr)); + Ort::ThrowOnError(ort_env->CopyTensor(input, gpu_input, nullptr)); auto outputs = sessions[thread_id]->Run(Ort::RunOptions{nullptr}, input_names.data(), &gpu_input, 1, output_names.data(), output_names.size()); std::array output_data{}; @@ -783,8 +776,8 @@ TEST_F(PluginEpWebGpuConcurrency, SharedGpuCopyIsSubmittedBeforeSessionRun) { kShape.data(), kShape.size()); auto source = Ort::Value::CreateTensor(allocator, kShape.data(), kShape.size()); auto destination = Ort::Value::CreateTensor(allocator, kShape.data(), kShape.size()); - ThrowOnError(ort_env->CopyTensor(input, source, nullptr)); - ThrowOnError(ort_env->CopyTensor(source, destination, nullptr)); + Ort::ThrowOnError(ort_env->CopyTensor(input, source, nullptr)); + Ort::ThrowOnError(ort_env->CopyTensor(source, destination, nullptr)); auto result = session->Run(Ort::RunOptions{nullptr}, input_names.data(), &destination, 1, output_names.data(), output_names.size()); std::array output{}; From 25ca2afeecb76ef3fbe8e427205256f01fa1ee87 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Fri, 2 Oct 2026 07:31:06 -0700 Subject: [PATCH 105/107] Add WebGPU EP GetDawnPlatform() known memory leak to debug_alloc.cc exceptions --- .../core/platform/windows/debug_alloc.cc | 75 +++++++++++-------- 1 file changed, 45 insertions(+), 30 deletions(-) diff --git a/onnxruntime/core/platform/windows/debug_alloc.cc b/onnxruntime/core/platform/windows/debug_alloc.cc index 14eb06a9cfa39..5107b720b1f5e 100644 --- a/onnxruntime/core/platform/windows/debug_alloc.cc +++ b/onnxruntime/core/platform/windows/debug_alloc.cc @@ -28,9 +28,13 @@ constexpr int c_callstack_limit = 32; // Maximum depth of callstack in leak tra #pragma warning(disable : 26490) // type.1 Don't use reinterpret_cast #pragma warning(disable : 26493) // type.4 Don't use C-style casts -#include -#include +#include +#include #include +#include +#include + +#include #include "debug_alloc.h" #include #pragma comment(lib, "Dbghelp.lib") @@ -228,34 +232,45 @@ Memory_LeakCheck::~Memory_LeakCheck() { const std::string string = message.str(); - // Google test has memory leaks that they haven't fixed. One such issue is tracked here: https://github.com/google/googletest/issues/692 - // - // In gtest-port.cc in function: static ThreadIdToThreadLocals* GetThreadLocalsMapLocked() - // static ThreadIdToThreadLocals* map = new ThreadIdToThreadLocals; - // - // In gtest-port.cc in Mutex::~Mutex() there is this comment: - // "Static mutexes are leaked intentionally. It is not thread-safe to try to clean them up." - // Which explains this leak inside of: void Mutex::ThreadSafeLazyInit() - // critical_section_ = new CRITICAL_SECTION; - // - // in google/re2 re2.cc initializes leaking singletons - // std::call_once(empty_once, []() { - // empty_string = new string; - // empty_named_groups = new std::map; - // empty_group_names = new std::map; }); - if (string.find("RtlRunOnceExecuteOnce") == std::string::npos && - string.find("re2::RE2::Init") == std::string::npos && - string.find("dynamic initializer for 'FLAGS_") == std::string::npos && - string.find("AbslFlagDefaultGenForgtest_") == std::string::npos && - string.find("AbslFlagDefaultGenForundefok::Gen") == std::string::npos && - string.find("::SetProgramUsageMessage") == std::string::npos && - string.find("testing::internal::ParseGoogleTestFlagsOnly") == std::string::npos && - string.find("testing::internal::Mutex::ThreadSafeLazyInit") == std::string::npos && - string.find("testing::internal::ThreadLocalRegistryImpl::GetThreadLocalsMapLocked") == std::string::npos && - string.find("testing::internal::ThreadLocalRegistryImpl::GetValueOnCurrentThread") == std::string::npos && - string.find("PyInit_onnxruntime_pybind11_state") == std::string::npos && - string.find("google::protobuf::internal::InitProtobufDefaultsSlow") == std::string::npos && - string.find("EtwEventWriteNoRegistration") == std::string::npos) { + constexpr auto allow_list = std::to_array({ + "RtlRunOnceExecuteOnce", + // In google/re2 re2.cc initializes leaking singletons + // std::call_once(empty_once, []() { + // empty_string = new string; + // empty_named_groups = new std::map; + // empty_group_names = new std::map; }); + "re2::RE2::Init", + "dynamic initializer for 'FLAGS_", + "AbslFlagDefaultGenForgtest_", + "AbslFlagDefaultGenForundefok::Gen", + "::SetProgramUsageMessage", + // Google test has memory leaks that they haven't fixed. One such issue is tracked here: + // https://github.com/google/googletest/issues/692 + // + // In gtest-port.cc in function: static ThreadIdToThreadLocals* GetThreadLocalsMapLocked() + // static ThreadIdToThreadLocals* map = new ThreadIdToThreadLocals; + // + // In gtest-port.cc in Mutex::~Mutex() there is this comment: + // "Static mutexes are leaked intentionally. It is not thread-safe to try to clean them up." + // Which explains this leak inside of: void Mutex::ThreadSafeLazyInit() + // critical_section_ = new CRITICAL_SECTION; + "testing::internal::ParseGoogleTestFlagsOnly", + "testing::internal::Mutex::ThreadSafeLazyInit", + "testing::internal::ThreadLocalRegistryImpl::GetThreadLocalsMapLocked", + "testing::internal::ThreadLocalRegistryImpl::GetValueOnCurrentThread", + "PyInit_onnxruntime_pybind11_state", + "google::protobuf::internal::InitProtobufDefaultsSlow", + "EtwEventWriteNoRegistration", +#if defined(USE_WEBGPU) + // In onnxruntime/core/providers/webgpu/webgpu_context_dawn_platform.cc:GetDawnPlatform(), there is an + // intentional leak of a function-local static: + // static DawnPlatform* platform = new DawnPlatform(); + "onnxruntime::webgpu::`anonymous namespace'::GetDawnPlatform", +#endif // defined(USE_WEBGPU), + }); + + if (std::none_of(allow_list.begin(), allow_list.end(), + [&string](std::string_view allowed) { return string.find(allowed) != std::string::npos; })) { if (leaked_bytes == 0) DebugPrint("\n-----Starting Heap Trace-----\n\n"); From d9cf06a26353faac5731cf76fde4bbc309655250 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Fri, 2 Oct 2026 19:53:52 -0700 Subject: [PATCH 106/107] revert binary size threshold update --- .github/workflows/android.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/android.yml b/.github/workflows/android.yml index a198cfb7d53a5..c5dbf786f7c0c 100644 --- a/.github/workflows/android.yml +++ b/.github/workflows/android.yml @@ -79,7 +79,7 @@ jobs: run: | set -e -x BINARY_SIZE_THRESHOLD_ARGS="" - BINARY_SIZE_THRESHOLD_IN_BYTES=1601536 + BINARY_SIZE_THRESHOLD_IN_BYTES=1599488 echo "Binary size threshold in bytes: ${BINARY_SIZE_THRESHOLD_IN_BYTES}" BINARY_SIZE_THRESHOLD_ARGS="--threshold_size_in_bytes ${BINARY_SIZE_THRESHOLD_IN_BYTES}" From 7e99529f03f7557ec2b961ac2a0ac8764471acd2 Mon Sep 17 00:00:00 2001 From: edgchen1 <18449977+edgchen1@users.noreply.github.com> Date: Fri, 2 Oct 2026 20:21:09 -0700 Subject: [PATCH 107/107] Revert "revert binary size threshold update" This reverts commit d9cf06a26353faac5731cf76fde4bbc309655250. --- .github/workflows/android.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/android.yml b/.github/workflows/android.yml index 8986b47867934..7ca48f5ee1881 100644 --- a/.github/workflows/android.yml +++ b/.github/workflows/android.yml @@ -79,7 +79,7 @@ jobs: run: | set -e -x BINARY_SIZE_THRESHOLD_ARGS="" - BINARY_SIZE_THRESHOLD_IN_BYTES=1599488 + BINARY_SIZE_THRESHOLD_IN_BYTES=1601536 echo "Binary size threshold in bytes: ${BINARY_SIZE_THRESHOLD_IN_BYTES}" BINARY_SIZE_THRESHOLD_ARGS="--threshold_size_in_bytes ${BINARY_SIZE_THRESHOLD_IN_BYTES}"