Skip to content

Add python implementation for wgsl-gen - #28355

Merged
guschmue merged 19 commits into
microsoft:mainfrom
danielsongmicrosoft:user/danielsong/templatelibrarypythonport
Jun 16, 2026
Merged

Add python implementation for wgsl-gen#28355
guschmue merged 19 commits into
microsoft:mainfrom
danielsongmicrosoft:user/danielsong/templatelibrarypythonport

Conversation

@danielsongmicrosoft

@danielsongmicrosoft danielsongmicrosoft commented May 4, 2026

Copy link
Copy Markdown
Contributor

Description

Replaces the npm-based @fs-eire/wgsl-template Node.js engine used by the WebGPU EP's static build path with an in-tree, stdlib-only Python implementation under wgsl_templates. The dynamic generator path is removed from the build surface - the code logic remains as unreachable and will be removed in a follow-up.

Motivation and Context

The WebGPU EP's WGSL template-to-C++ codegen is currently driven by the @fs-eire/wgsl-template npm package, so any build that touches a .wgsl.template file needs Node.js installed and has to pull the package over the network. This npm package is also no longer maintained.

@danielsongmicrosoft
danielsongmicrosoft marked this pull request as draft May 4, 2026 23:23
@guschmue guschmue added the ep:WebGPU ort-web webgpu provider label May 5, 2026
@guschmue

guschmue commented May 5, 2026

Copy link
Copy Markdown
Contributor

I think this should move to tools/python/

@danielsongmicrosoft

danielsongmicrosoft commented May 5, 2026

Copy link
Copy Markdown
Contributor Author

Did some manual testing for confirmation:

  • Built ORT with --use_webgpu --wgsl_template static --config RelWithDebInfo
    • I believe this exercises the new py tool against all 27 in-tree .wgsl.template files and c++ compiler accepted it
  • Ran onnxruntime_provider_test.exe with GTX 1060 6GB (what I have on my dev box) for Pad, Transpose, and Conv operator suites.
    • 317/318 tests passed, the 1 failure was due to an fp16 compatibility issue with my GPU

@danielsongmicrosoft

Copy link
Copy Markdown
Contributor Author

@danielsongmicrosoft please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

  • (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.
@microsoft-github-policy-service agree
  • (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.
@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree company="Microsoft"

Comment thread tools/python/wgsl_template/generator.py Fixed
Comment thread tools/python/wgsl_template/generator.py Fixed
Comment thread tools/python/wgsl_template/generator.py Fixed
@danielsongmicrosoft
danielsongmicrosoft marked this pull request as ready for review May 28, 2026 22:00
Comment thread cmake/onnxruntime_providers_webgpu.cmake Outdated
Comment thread cmake/onnxruntime_providers_webgpu.cmake Outdated
Comment thread tools/python/wgsl_template/test/test_build.py Outdated
Comment thread tools/python/wgsl_template/test/test_build.py Outdated
@danielsongmicrosoft
danielsongmicrosoft force-pushed the user/danielsong/templatelibrarypythonport branch from c6505c0 to d0943f8 Compare June 4, 2026 23:10
@guschmue
guschmue requested a review from Copilot June 5, 2026 15:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR replaces the WebGPU EP’s npm/Node.js-based WGSL template generator (@fs-eire/wgsl-template) with an in-tree, stdlib-only Python implementation (PASS0 loader / PASS1 parser / PASS2 generator + C++ emitters). It also removes the dynamic WGSL generator option from the supported build surface (leaving the old dynamic path in C++ as unreachable with TODOs), updates CMake/build tooling to invoke the Python CLI, and adds a comprehensive Python unit+fixture test suite (including an in-tree smoke test over the real WebGPU templates).

Changes:

  • Add a new tools/python/wgsl_template/ package implementing the WGSL template pipeline and static C++ codegen.
  • Switch WebGPU CMake generation from npm to Python (tools/python/wgsl_gen.py) and remove --wgsl_template build options/plumbing.
  • Add Python tests + golden fixtures for loader/parser/generator/build behavior and an in-tree template smoke test.

Reviewed changes

Copilot reviewed 145 out of 151 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
tools/python/wgsl_template/types.py New shared dataclasses for PASS0/1/2 and metadata.
tools/python/wgsl_template/errors.py New error hierarchy for loader/parser/generator/build.
tools/python/wgsl_template/loader.py PASS0: directory scanning, path normalization, security checks.
tools/python/wgsl_template/parser.py PASS1: comment stripping, #include, #define handling.
tools/python/wgsl_template/code_pattern.py PASS2 patterns and #use pattern registry.
tools/python/wgsl_template/generator.py PASS2 generator state machine and validation.
tools/python/wgsl_template/code_generator/init.py Code generator resolver (static-cpp*).
tools/python/wgsl_template/code_generator/static_cpp.py Static C++ and string-table emit implementation.
tools/python/wgsl_template/init.py Public build() orchestrator + safe/idempotent file writes.
tools/python/wgsl_gen.py CLI entry point used by CMake for template generation.
tools/python/wgsl_template/test/run_tests.py Aggregated unittest discovery runner.
tools/python/wgsl_template/test/test_loader.py PASS0-focused unit tests.
tools/python/wgsl_template/test/test_parser.py PASS1-focused + fixture-driven tests.
tools/python/wgsl_template/test/test_generator.py PASS2-focused + fixture-driven tests.
tools/python/wgsl_template/test/test_build.py End-to-end build() tests + fixture runner.
tools/python/wgsl_template/test/test_in_tree_smoke.py Smoke tests over real in-tree WebGPU templates.
tools/python/wgsl_template/test/testcases/.gitattributes Enforce LF for golden/expected files.
tools/python/wgsl_template/test/testcases/parser-basic/test-config.json Parser fixture config.
tools/python/wgsl_template/test/testcases/parser-basic/basic.wgsl.template Parser fixture input.
tools/python/wgsl_template/test/testcases/parser-basic/basic.wgsl.template.pass1 Parser golden output.
tools/python/wgsl_template/test/testcases/parser-comment-removal/test-config.json Comment-stripping fixture config.
tools/python/wgsl_template/test/testcases/parser-comment-removal/comments.wgsl.template Comment-stripping fixture input.
tools/python/wgsl_template/test/testcases/parser-comment-removal/comments.wgsl.template.pass1 Comment-stripping golden output.
tools/python/wgsl_template/test/testcases/parser-empty/test-config.json Empty-dir parser fixture config.
tools/python/wgsl_template/test/testcases/parser-include-basic/test-config.json Include fixture config.
tools/python/wgsl_template/test/testcases/parser-include-basic/main.wgsl.template Include fixture input.
tools/python/wgsl_template/test/testcases/parser-include-basic/main.wgsl.template.pass1 Include golden output.
tools/python/wgsl_template/test/testcases/parser-include-basic/utils.wgsl.template Included fixture input.
tools/python/wgsl_template/test/testcases/parser-include-basic/utils.wgsl.template.pass1 Included golden output.
tools/python/wgsl_template/test/testcases/parser-include-inside-comment/test-config.json “Include in comments ignored” fixture config.
tools/python/wgsl_template/test/testcases/parser-include-inside-comment/main.wgsl.template Fixture input.
tools/python/wgsl_template/test/testcases/parser-include-inside-comment/main.wgsl.template.pass1 Golden output.
tools/python/wgsl_template/test/testcases/parser-include-inside-comment/utils.wgsl.template Fixture input.
tools/python/wgsl_template/test/testcases/parser-include-inside-comment/utils.wgsl.template.pass1 Golden output.
tools/python/wgsl_template/test/testcases/parser-include-invalid-empty/test-config.json Invalid-include fixture config.
tools/python/wgsl_template/test/testcases/parser-include-invalid-empty/main.wgsl.template Invalid-include fixture input.
tools/python/wgsl_template/test/testcases/parser-include-invalid-no-quotes/test-config.json Invalid-include fixture config.
tools/python/wgsl_template/test/testcases/parser-include-invalid-no-quotes/main.wgsl.template Invalid-include fixture input.
tools/python/wgsl_template/test/testcases/parser-include-invalid-nonexist/test-config.json Missing-include fixture config.
tools/python/wgsl_template/test/testcases/parser-include-invalid-nonexist/main.wgsl.template Missing-include fixture input.
tools/python/wgsl_template/test/testcases/parser-include-invalid-path/test-config.json Invalid-path include fixture config.
tools/python/wgsl_template/test/testcases/parser-include-invalid-path/main.wgsl.template Invalid-path include fixture input.
tools/python/wgsl_template/test/testcases/parser-include-invalid-unclosed-quotes/test-config.json Unclosed-quote include fixture config.
tools/python/wgsl_template/test/testcases/parser-include-invalid-unclosed-quotes/main.wgsl.template Unclosed-quote include fixture input.
tools/python/wgsl_template/test/testcases/parser-include-multi-layer/test-config.json Nested-include fixture config.
tools/python/wgsl_template/test/testcases/parser-include-multi-layer/main.wgsl.template Nested-include fixture input.
tools/python/wgsl_template/test/testcases/parser-include-multi-layer/main.wgsl.template.pass1 Nested-include golden output.
tools/python/wgsl_template/test/testcases/parser-include-multi-layer/graphics.wgsl.template Nested-include fixture input.
tools/python/wgsl_template/test/testcases/parser-include-multi-layer/graphics.wgsl.template.pass1 Nested-include golden output.
tools/python/wgsl_template/test/testcases/parser-include-multi-layer/math.wgsl.template Nested-include fixture input.
tools/python/wgsl_template/test/testcases/parser-include-multi-layer/math.wgsl.template.pass1 Nested-include golden output.
tools/python/wgsl_template/test/testcases/parser-include-multi-layer/constants.wgsl.template Nested-include fixture input.
tools/python/wgsl_template/test/testcases/parser-include-multi-layer/constants.wgsl.template.pass1 Nested-include golden output.
tools/python/wgsl_template/test/testcases/parser-include-space-prefix/test-config.json Leading-space include fixture config.
tools/python/wgsl_template/test/testcases/parser-include-space-prefix/main.wgsl.template Leading-space include fixture input.
tools/python/wgsl_template/test/testcases/parser-include-space-prefix/main.wgsl.template.pass1 Leading-space include golden output.
tools/python/wgsl_template/test/testcases/parser-include-space-prefix/constants.wgsl.template Included fixture input.
tools/python/wgsl_template/test/testcases/parser-include-space-prefix/constants.wgsl.template.pass1 Included golden output.
tools/python/wgsl_template/test/testcases/parser-macro-basic/test-config.json Macro fixture config.
tools/python/wgsl_template/test/testcases/parser-macro-basic/basic.wgsl.template Macro fixture input.
tools/python/wgsl_template/test/testcases/parser-macro-basic/basic.wgsl.template.pass1 Macro golden output.
tools/python/wgsl_template/test/testcases/parser-macro-chain/test-config.json Macro chaining fixture config.
tools/python/wgsl_template/test/testcases/parser-macro-chain/chain.wgsl.template Macro chaining fixture input.
tools/python/wgsl_template/test/testcases/parser-macro-chain/chain.wgsl.template.pass1 Macro chaining golden output.
tools/python/wgsl_template/test/testcases/parser-macro-complex-values/test-config.json Complex macro value fixture config.
tools/python/wgsl_template/test/testcases/parser-macro-complex-values/complex.wgsl.template Complex macro value fixture input.
tools/python/wgsl_template/test/testcases/parser-macro-complex-values/complex.wgsl.template.pass1 Complex macro value golden output.
tools/python/wgsl_template/test/testcases/parser-macro-invalid-circular/test-config.json Circular macro fixture config.
tools/python/wgsl_template/test/testcases/parser-macro-invalid-circular/circular.wgsl.template Circular macro fixture input.
tools/python/wgsl_template/test/testcases/parser-macro-invalid-duplicate/test-config.json Duplicate macro fixture config.
tools/python/wgsl_template/test/testcases/parser-macro-invalid-duplicate/duplicate.wgsl.template Duplicate macro fixture input.
tools/python/wgsl_template/test/testcases/parser-macro-invalid-empty-value/test-config.json Empty macro fixture config.
tools/python/wgsl_template/test/testcases/parser-macro-invalid-empty-value/empty.wgsl.template Empty macro fixture input.
tools/python/wgsl_template/test/testcases/parser-macro-invalid-syntax/test-config.json Invalid macro name fixture config.
tools/python/wgsl_template/test/testcases/parser-macro-invalid-syntax/invalid.wgsl.template Invalid macro name fixture input.
tools/python/wgsl_template/test/testcases/parser-macro-invalid-whitespace-value/test-config.json Whitespace-only macro value fixture config.
tools/python/wgsl_template/test/testcases/parser-macro-invalid-whitespace-value/whitespace.wgsl.template Whitespace-only macro value fixture input.
tools/python/wgsl_template/test/testcases/parser-macro-name-edge-cases/test-config.json Macro name edge-cases fixture config.
tools/python/wgsl_template/test/testcases/parser-macro-name-edge-cases/names.wgsl.template Macro name edge-cases fixture input.
tools/python/wgsl_template/test/testcases/parser-macro-name-edge-cases/names.wgsl.template.pass1 Macro name edge-cases golden output.
tools/python/wgsl_template/test/testcases/parser-macro-space-prefix/test-config.json Leading-space macro fixture config.
tools/python/wgsl_template/test/testcases/parser-macro-space-prefix/main.wgsl.template Leading-space macro fixture input.
tools/python/wgsl_template/test/testcases/parser-macro-space-prefix/main.wgsl.template.pass1 Leading-space macro golden output.
tools/python/wgsl_template/test/testcases/parser-subfolder/test-config.json Subfolder include fixture config.
tools/python/wgsl_template/test/testcases/parser-subfolder/main.wgsl.template Subfolder fixture input.
tools/python/wgsl_template/test/testcases/parser-subfolder/main.wgsl.template.pass1 Subfolder golden output.
tools/python/wgsl_template/test/testcases/parser-subfolder/shaders/vertex.wgsl.template Subfolder fixture input.
tools/python/wgsl_template/test/testcases/parser-subfolder/shaders/vertex.wgsl.template.pass1 Subfolder golden output.
tools/python/wgsl_template/test/testcases/parser-subfolder/utils/common.wgsl.template Subfolder fixture input.
tools/python/wgsl_template/test/testcases/parser-subfolder/utils/common.wgsl.template.pass1 Subfolder golden output.
tools/python/wgsl_template/test/testcases/generator-basic/test-config.json Generator fixture config.
tools/python/wgsl_template/test/testcases/generator-basic/test.wgsl.template Generator fixture input.
tools/python/wgsl_template/test/testcases/generator-basic/test.wgsl.template.static-cpp-literal.gen Generator golden output.
tools/python/wgsl_template/test/testcases/generator-preserve-code-ref/test-config.json Preserve-code-ref generator fixture config.
tools/python/wgsl_template/test/testcases/generator-preserve-code-ref/test.wgsl.template Preserve-code-ref fixture input.
tools/python/wgsl_template/test/testcases/generator-preserve-code-ref/test.wgsl.template.static-cpp-literal.gen Preserve-code-ref golden output.
tools/python/wgsl_template/test/testcases/generator-param-basic/test-config.json Param substitution generator fixture config.
tools/python/wgsl_template/test/testcases/generator-param-basic/compute-param.wgsl.template Param substitution fixture input.
tools/python/wgsl_template/test/testcases/generator-param-basic/compute-param.wgsl.template.static-cpp-literal.gen Param substitution golden output.
tools/python/wgsl_template/test/testcases/generator-param-basic/compute-no-param.wgsl.template Param substitution fixture input.
tools/python/wgsl_template/test/testcases/generator-param-basic/compute-no-param.wgsl.template.static-cpp-literal.gen Param substitution golden output.
tools/python/wgsl_template/test/testcases/generator-main/test-config.json $MAIN generator fixture config.
tools/python/wgsl_template/test/testcases/generator-main/main.wgsl.template $MAIN fixture input.
tools/python/wgsl_template/test/testcases/generator-main/main.wgsl.template.static-cpp-literal.gen $MAIN golden output.
tools/python/wgsl_template/test/testcases/generator-main-invalid-unclosed/test-config.json Unclosed $MAIN fixture config.
tools/python/wgsl_template/test/testcases/generator-main-invalid-unclosed/test.wgsl.template Unclosed $MAIN fixture input.
tools/python/wgsl_template/test/testcases/generator-main-invalid-syntax/test-config.json Invalid $MAIN syntax fixture config.
tools/python/wgsl_template/test/testcases/generator-main-invalid-syntax/test.wgsl.template Invalid $MAIN syntax fixture input.
tools/python/wgsl_template/test/testcases/generator-main-invalid-nested/test-config.json Nested $MAIN fixture config.
tools/python/wgsl_template/test/testcases/generator-main-invalid-nested/test.wgsl.template Nested $MAIN fixture input.
tools/python/wgsl_template/test/testcases/generator-main-invalid-multiple/test-config.json Multiple $MAIN fixture config.
tools/python/wgsl_template/test/testcases/generator-main-invalid-multiple/test.wgsl.template Multiple $MAIN fixture input.
tools/python/wgsl_template/test/testcases/generator-example-pad/test-config.json Complex generator fixture config.
tools/python/wgsl_template/test/testcases/generator-example-pad/pad.wgsl.template Complex generator fixture input.
tools/python/wgsl_template/test/testcases/generator-example-pad/pad.wgsl.template.static-cpp-literal.gen Complex generator golden output.
tools/python/wgsl_template/test/testcases/build-basic/test-config.json Build fixture config.
tools/python/wgsl_template/test/testcases/build-basic/src/shader/triangle.wgsl.template Build fixture input template.
tools/python/wgsl_template/test/testcases/build-basic/expected/static-cpp/index.h Build fixture expected header index.
tools/python/wgsl_template/test/testcases/build-basic/expected/static-cpp/index_impl.h Build fixture expected impl index.
tools/python/wgsl_template/test/testcases/build-basic/expected/static-cpp/string_table.h Build fixture expected string table.
tools/python/wgsl_template/test/testcases/build-basic/expected/static-cpp/generated/shader/triangle.h Build fixture expected per-template header.
tools/python/wgsl_template/test/testcases/build-basic/expected/dynamic/templates.js Legacy expected artifact (not exercised by new harness).
tools/python/wgsl_template/test/testcases/build-example-pad/test-config.json Build fixture config.
tools/python/wgsl_template/test/testcases/build-example-pad/src/tensor/pad.wgsl.template Build fixture input template.
tools/python/wgsl_template/test/testcases/build-example-pad/expected/static-cpp-literal/index.h Build fixture expected header index.
tools/python/wgsl_template/test/testcases/build-example-pad/expected/static-cpp-literal/index_impl.h Build fixture expected impl index.
tools/python/wgsl_template/test/testcases/build-example-pad/expected/static-cpp-literal/generated/tensor/pad.h Build fixture expected per-template header.
tools/python/wgsl_template/test/testcases/build-example-pad/expected/dynamic/templates.js Legacy expected artifact (not exercised by new harness).
tools/python/wgsl_template/test/testcases/build-directories/test-config.json Multi-source-dir build fixture config.
tools/python/wgsl_template/test/testcases/build-directories/src/common/utils.wgsl.template Build fixture input template.
tools/python/wgsl_template/test/testcases/build-directories/src/effects/blur.wgsl.template Build fixture input template.
tools/python/wgsl_template/test/testcases/build-directories/src/effects/lighting.wgsl.template Build fixture input template.
tools/python/wgsl_template/test/testcases/build-directories/expected/static-cpp/index.h Build fixture expected header index.
tools/python/wgsl_template/test/testcases/build-directories/expected/static-cpp/index_impl.h Build fixture expected impl index.
tools/python/wgsl_template/test/testcases/build-directories/expected/static-cpp/string_table.h Build fixture expected string table.
tools/python/wgsl_template/test/testcases/build-directories/expected/static-cpp/generated/utils.h Build fixture expected per-template header.
tools/python/wgsl_template/test/testcases/build-directories/expected/static-cpp/generated/@effects/blur.h Build fixture expected per-template header.
tools/python/wgsl_template/test/testcases/build-directories/expected/static-cpp/generated/@effects/lighting.h Build fixture expected per-template header.
tools/ci_build/build_args.py Remove --wgsl_template CLI option.
tools/ci_build/build.py Remove WGSL template generator CMake flag/vcpkg feature plumbing.
cmake/CMakeLists.txt Remove onnxruntime_WGSL_TEMPLATE option.
cmake/onnxruntime_providers_webgpu.cmake Switch WGSL generation from npm to Python + add ctest hook.
cmake/external/onnxruntime_external_deps.cmake Note dynamic duktape path is now unreachable.
cmake/deps.txt Note duktape planned removal once dynamic generator is deleted.
onnxruntime/core/providers/webgpu/wgsl_templates/wgsl_gen.h Add TODO noting dynamic path now unreachable.
onnxruntime/core/providers/webgpu/wgsl_templates/wgsl_gen.cc Add TODO noting dynamic path now unreachable.
onnxruntime/core/providers/webgpu/wgsl_templates/README.md Update docs to Python generator + remove dynamic-mode guidance.
onnxruntime/core/providers/webgpu/wgsl_templates/package.json Remove Node.js dependency manifest.
onnxruntime/core/providers/webgpu/wgsl_templates/package-lock.json Remove npm lockfile.
onnxruntime/core/providers/webgpu/wgsl_templates/.gitignore Remove node_modules ignore.
.github/workflows/windows_webgpu.yml Remove wgsl_template matrix/flag usage.
Files not reviewed (1)
  • onnxruntime/core/providers/webgpu/wgsl_templates/package-lock.json: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tools/python/wgsl_template/parser.py
Comment thread tools/python/wgsl_template/loader.py
Comment thread tools/python/wgsl_template/generator.py
Comment thread tools/python/wgsl_template/code_generator/static_cpp.py
Comment thread cmake/external/onnxruntime_external_deps.cmake Outdated
Comment thread tools/python/wgsl_template/code_generator/__init__.py
Comment thread tools/python/wgsl_gen.py Outdated
Comment thread tools/python/wgsl_gen.py Outdated
Comment thread tools/python/wgsl_template/parser.py Outdated
Comment thread tools/python/wgsl_template/test/run_tests.py
Comment thread tools/python/wgsl_template/test/test_build.py Outdated
Comment thread tools/python/wgsl_template/test/test_in_tree_smoke.py Outdated
Comment thread tools/python/wgsl_template/test/testcases/generator-main/test-config.json Outdated
@guschmue guschmue closed this Jun 15, 2026
@guschmue guschmue reopened this Jun 15, 2026
guschmue
guschmue previously approved these changes Jun 15, 2026
@danielsongmicrosoft
danielsongmicrosoft force-pushed the user/danielsong/templatelibrarypythonport branch from dca9d9c to 549a11a Compare June 15, 2026 23:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ep:WebGPU ort-web webgpu provider

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants