-
Notifications
You must be signed in to change notification settings - Fork 4.1k
Add webgpu plugin EP pipeline #27841
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 17 commits
Commits
Show all changes
31 commits
Select commit
Hold shift + click to select a range
9013904
webgpu plugin pipeline
fs-eire 23782b9
2
fs-eire 0a8b240
linux
fs-eire f7c6c34
change npm feed
fs-eire 4639345
fix linux build 2
fs-eire 8cd1590
include version info
fs-eire 794c6a3
fix build tool path
fs-eire 1492f57
fix linux build 3
fs-eire ad6d885
exclude build tools from code sign
fs-eire 781627e
dxc copy
fs-eire 7334ea9
publish U package
fs-eire fe278cd
validate params
fs-eire e4f654d
use version string to detect ORT API version
fs-eire f6d38f5
update packaging pipeline
fs-eire 113c823
fix build
fs-eire 1b3cee6
resolve comments
fs-eire 9037880
Merge remote-tracking branch 'origin/main' into fs-eire/webgpu-plugin…
fs-eire 2a1ffff
fix build flag for arm64
fs-eire 31c4f4c
apply fix according to comments
fs-eire 4b97ea1
save current ort api version
fs-eire 85eb070
resolve comments
fs-eire 04278d9
fix build
fs-eire a81e76d
resolve comments
fs-eire 4d8a762
Merge remote-tracking branch 'origin/main' into fs-eire/webgpu-plugin…
fs-eire ea3aa72
resolve comments
fs-eire 92bfc30
fix build
fs-eire 1214dd2
suppress MSVC ICE
fs-eire 70174de
Address PR review feedback: null checks, binary verification, and nits
fs-eire 1c90606
Add DXC hash verification and fail on RC versioning
fs-eire 1b868f3
Update tools/ci_build/github/linux/build_webgpu_plugin_package.sh
fs-eire e0a4ce8
sign dxc dlls
fs-eire File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
117 changes: 117 additions & 0 deletions
117
tools/ci_build/github/azure-pipelines/plugin-webgpu-pipeline.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,117 @@ | ||
| trigger: none | ||
|
|
||
| resources: | ||
| repositories: | ||
| - repository: 1esPipelines | ||
| type: git | ||
| name: 1ESPipelineTemplates/1ESPipelineTemplates | ||
| ref: refs/tags/release | ||
|
|
||
| parameters: | ||
| - name: build_windows_x64 | ||
| displayName: 'Build Windows x64' | ||
| type: boolean | ||
| default: true | ||
|
|
||
| - name: build_windows_arm64 | ||
| displayName: 'Build Windows ARM64' | ||
| type: boolean | ||
| default: false | ||
|
|
||
| - name: build_linux_x64 | ||
| displayName: 'Build Linux x64' | ||
| type: boolean | ||
| default: false | ||
|
|
||
| - name: build_macos_arm64 | ||
| displayName: 'Build macOS ARM64' | ||
| type: boolean | ||
| default: false | ||
|
|
||
| - name: package_version | ||
| displayName: 'Package Version' | ||
| type: string | ||
| values: | ||
| - release | ||
| - RC | ||
| - dev | ||
| default: dev | ||
|
|
||
| - name: cmake_build_type | ||
| type: string | ||
| default: 'Release' | ||
| values: | ||
| - Debug | ||
| - Release | ||
| - RelWithDebInfo | ||
| - MinSizeRel | ||
|
|
||
| variables: | ||
| # Windows ARM64 build requires Windows x64 build to be enabled (ARM64 cross-compilation depends on x64 build artifacts) | ||
| - name: invalidARM64Config | ||
| value: ${{ and(eq(parameters.build_windows_arm64, true), eq(parameters.build_windows_x64, false)) }} | ||
| # Non-dev package versions (release, RC) must use Release build type | ||
| - name: invalidBuildTypeConfig | ||
| value: ${{ and(ne(parameters.package_version, 'dev'), ne(parameters.cmake_build_type, 'Release')) }} | ||
|
|
||
| extends: | ||
| template: v1/1ES.Official.PipelineTemplate.yml@1esPipelines | ||
| parameters: | ||
| settings: | ||
| networkIsolationPolicy: Permissive | ||
| sdl: | ||
| componentgovernance: | ||
| ignoreDirectories: '$(Build.Repository.LocalPath)/cmake/external/emsdk/upstream/emscripten/tests,$(Build.Repository.LocalPath)/cmake/external/onnx/third_party/benchmark,$(Build.Repository.LocalPath)/cmake/external/onnx/third_party/pybind11,$(Build.Repository.LocalPath)/cmake/external/onnx/third_party/pybind11/tests,$(Build.Repository.LocalPath)/cmake/external/onnxruntime-extensions,$(Build.Repository.LocalPath)/js/react_native/e2e/node_modules,$(Build.Repository.LocalPath)/js/node_modules,$(Build.Repository.LocalPath)/onnxruntime-inference-examples,$(Build.SourcesDirectory)/cmake/external/emsdk/upstream/emscripten/tests,$(Build.SourcesDirectory)/cmake/external/onnx/third_party/benchmark,$(Build.SourcesDirectory)/cmake/external/onnx/third_party/pybind11,$(Build.SourcesDirectory)/cmake/external/onnx/third_party/pybind11/tests,$(Build.SourcesDirectory)/cmake/external/onnxruntime-extensions,$(Build.SourcesDirectory)/js/react_native/e2e/node_modules,$(Build.SourcesDirectory)/js/node_modules,$(Build.SourcesDirectory)/onnxruntime-inference-examples,$(Build.BinariesDirectory)' | ||
| alertWarningLevel: High | ||
| failOnAlert: false | ||
| verbosity: Normal | ||
| timeout: 3600 | ||
| tsa: | ||
| enabled: true | ||
| codeSignValidation: | ||
| enabled: true | ||
| break: true | ||
| policheck: | ||
| enabled: true | ||
| exclusionsFile: '$(Build.SourcesDirectory)\tools\ci_build\policheck_exclusions.xml' | ||
| codeql: | ||
| compiled: | ||
| enabled: false | ||
| justificationForDisabling: 'CodeQL is taking nearly 6 hours resulting in timeouts in our production pipelines' | ||
| pool: | ||
| name: 'onnxruntime-Win-CPU-VS2022-Latest' | ||
| os: windows | ||
|
|
||
| stages: | ||
| # Validate parameter combinations | ||
| - ${{ if or(eq(variables['invalidARM64Config'], 'True'), eq(variables['invalidBuildTypeConfig'], 'True')) }}: | ||
| - stage: Validate_Parameters | ||
| displayName: 'Validate Parameters' | ||
| dependsOn: [] | ||
| jobs: | ||
| - job: Validate | ||
| displayName: 'Validate parameter combinations' | ||
| pool: | ||
| name: 'onnxruntime-Win-CPU-VS2022-Latest' | ||
| os: windows | ||
| steps: | ||
| - checkout: none | ||
| - ${{ if eq(variables['invalidARM64Config'], 'True') }}: | ||
| - script: | | ||
| echo "##vso[task.logissue type=error]Windows ARM64 build requires Windows x64 build to be enabled." | ||
| exit 1 | ||
| displayName: 'ERROR: Windows ARM64 requires Windows x64' | ||
| - ${{ if eq(variables['invalidBuildTypeConfig'], 'True') }}: | ||
| - script: | | ||
| echo "##vso[task.logissue type=error]Non-dev package version requires Release build type." | ||
| exit 1 | ||
| displayName: 'ERROR: Non-dev package version requires Release build type' | ||
| - ${{ else }}: | ||
| - template: stages/plugin-webgpu-packaging-stage.yml | ||
| parameters: | ||
| build_windows_x64: ${{ parameters.build_windows_x64 }} | ||
| build_windows_arm64: ${{ parameters.build_windows_arm64 }} | ||
| build_linux_x64: ${{ parameters.build_linux_x64 }} | ||
| build_macos_arm64: ${{ parameters.build_macos_arm64 }} | ||
| package_version: ${{ parameters.package_version }} | ||
| cmake_build_type: ${{ parameters.cmake_build_type }} |
96 changes: 96 additions & 0 deletions
96
tools/ci_build/github/azure-pipelines/stages/plugin-linux-webgpu-stage.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,96 @@ | ||
| parameters: | ||
| - name: machine_pool | ||
| type: string | ||
| default: 'onnxruntime-Ubuntu2404-AMD-CPU' | ||
|
|
||
| - name: package_version | ||
| type: string | ||
| default: dev | ||
|
|
||
| - name: cmake_build_type | ||
| type: string | ||
| default: 'Release' | ||
| values: | ||
| - Debug | ||
| - Release | ||
| - RelWithDebInfo | ||
| - MinSizeRel | ||
|
|
||
| - name: docker_base_image | ||
| type: string | ||
| default: 'onnxruntimebuildcache.azurecr.io/internal/azureml/onnxruntime/build/cuda12_x64_almalinux8_gcc14:20251017.1' | ||
|
|
||
| stages: | ||
| - stage: Linux_plugin_webgpu_x64_Build | ||
| dependsOn: [] | ||
| jobs: | ||
| - job: Linux_plugin_webgpu_x64_Build | ||
| timeoutInMinutes: 240 | ||
| workspace: | ||
| clean: all | ||
| pool: | ||
| name: ${{ parameters.machine_pool }} | ||
| os: linux | ||
| templateContext: | ||
| outputs: | ||
| - output: pipelineArtifact | ||
| targetPath: $(Build.ArtifactStagingDirectory) | ||
| artifactName: webgpu_plugin_linux_x64 | ||
| variables: | ||
| - template: ../templates/common-variables.yml | ||
| steps: | ||
| - checkout: self | ||
| clean: true | ||
| submodules: recursive | ||
|
|
||
| - template: ../templates/set-nightly-build-option-variable-step.yml | ||
|
|
||
| - template: ../templates/set-plugin-build-variables-step.yml | ||
| parameters: | ||
| package_version: ${{ parameters.package_version }} | ||
|
|
||
| - template: ../templates/get-docker-image-steps.yml | ||
| parameters: | ||
| Dockerfile: tools/ci_build/github/linux/docker/inference/x86_64/python/cuda/Dockerfile | ||
| Context: tools/ci_build/github/linux/docker/inference/x86_64/python/cuda | ||
| DockerBuildArgs: "--build-arg BASEIMAGE=${{ parameters.docker_base_image }} --build-arg BUILD_UID=$( id -u )" | ||
| Repository: onnxruntimewebgpuplugin | ||
|
|
||
| - script: $(Build.SourcesDirectory)/tools/ci_build/github/linux/build_webgpu_plugin_package.sh -i onnxruntimewebgpuplugin -c ${{ parameters.cmake_build_type }} | ||
| workingDirectory: $(Build.SourcesDirectory) | ||
| displayName: 'Build WebGPU Plugin' | ||
| env: | ||
| EXTRA_CMAKE_DEFINES: $(PluginEpVersionDefine) | ||
|
|
||
| - script: | | ||
| set -e -x | ||
| mkdir -p $(Build.ArtifactStagingDirectory)/bin | ||
| plugin_path="$(Build.BinariesDirectory)/${{ parameters.cmake_build_type }}/libonnxruntime_providers_webgpu.so" | ||
| if [ ! -f "$plugin_path" ]; then | ||
| echo "Error: Expected plugin binary not found at '$plugin_path'. Failing build to avoid publishing an invalid package." | ||
| exit 1 | ||
| fi | ||
| cp "$plugin_path" "$(Build.ArtifactStagingDirectory)/bin/" | ||
| displayName: 'Copy plugin binaries' | ||
|
|
||
| - script: | | ||
| set -e -x | ||
| mkdir -p "$(Build.ArtifactStagingDirectory)/version" | ||
| touch "$(Build.ArtifactStagingDirectory)/version/$(PluginPackageVersion)" | ||
| displayName: 'Create version marker file' | ||
|
|
||
| - script: | | ||
| set -e -x | ||
| mkdir -p "$(Build.BinariesDirectory)/universal_package" | ||
| cp -R "$(Build.ArtifactStagingDirectory)/bin/"* "$(Build.BinariesDirectory)/universal_package/" | ||
| displayName: 'Stage binaries for universal package' | ||
|
|
||
| - task: UniversalPackages@0 | ||
| displayName: 'Publish universal package' | ||
| inputs: | ||
| command: publish | ||
| publishDirectory: '$(Build.BinariesDirectory)/universal_package' | ||
| vstsFeedPublish: 'PublicPackages/ORT-Nightly' | ||
| vstsFeedPackagePublish: 'onnxruntime-plugin-ep-webgpu-linux-x64' | ||
| versionOption: custom | ||
| versionPublish: '$(PluginUniversalPackageVersion)' |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.