Add python implementation for wgsl-gen - #28355
Conversation
|
I think this should move to tools/python/ |
|
Did some manual testing for confirmation:
|
@microsoft-github-policy-service agree company="Microsoft" |
c6505c0 to
d0943f8
Compare
There was a problem hiding this comment.
Pull request overview
This PR replaces the WebGPU EP’s npm/Node.js-based WGSL template generator (@fs-eire/wgsl-template) with an in-tree, stdlib-only Python implementation (PASS0 loader / PASS1 parser / PASS2 generator + C++ emitters). It also removes the dynamic WGSL generator option from the supported build surface (leaving the old dynamic path in C++ as unreachable with TODOs), updates CMake/build tooling to invoke the Python CLI, and adds a comprehensive Python unit+fixture test suite (including an in-tree smoke test over the real WebGPU templates).
Changes:
- Add a new
tools/python/wgsl_template/package implementing the WGSL template pipeline and static C++ codegen. - Switch WebGPU CMake generation from npm to Python (
tools/python/wgsl_gen.py) and remove--wgsl_templatebuild options/plumbing. - Add Python tests + golden fixtures for loader/parser/generator/build behavior and an in-tree template smoke test.
Reviewed changes
Copilot reviewed 145 out of 151 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/python/wgsl_template/types.py | New shared dataclasses for PASS0/1/2 and metadata. |
| tools/python/wgsl_template/errors.py | New error hierarchy for loader/parser/generator/build. |
| tools/python/wgsl_template/loader.py | PASS0: directory scanning, path normalization, security checks. |
| tools/python/wgsl_template/parser.py | PASS1: comment stripping, #include, #define handling. |
| tools/python/wgsl_template/code_pattern.py | PASS2 patterns and #use pattern registry. |
| tools/python/wgsl_template/generator.py | PASS2 generator state machine and validation. |
| tools/python/wgsl_template/code_generator/init.py | Code generator resolver (static-cpp*). |
| tools/python/wgsl_template/code_generator/static_cpp.py | Static C++ and string-table emit implementation. |
| tools/python/wgsl_template/init.py | Public build() orchestrator + safe/idempotent file writes. |
| tools/python/wgsl_gen.py | CLI entry point used by CMake for template generation. |
| tools/python/wgsl_template/test/run_tests.py | Aggregated unittest discovery runner. |
| tools/python/wgsl_template/test/test_loader.py | PASS0-focused unit tests. |
| tools/python/wgsl_template/test/test_parser.py | PASS1-focused + fixture-driven tests. |
| tools/python/wgsl_template/test/test_generator.py | PASS2-focused + fixture-driven tests. |
| tools/python/wgsl_template/test/test_build.py | End-to-end build() tests + fixture runner. |
| tools/python/wgsl_template/test/test_in_tree_smoke.py | Smoke tests over real in-tree WebGPU templates. |
| tools/python/wgsl_template/test/testcases/.gitattributes | Enforce LF for golden/expected files. |
| tools/python/wgsl_template/test/testcases/parser-basic/test-config.json | Parser fixture config. |
| tools/python/wgsl_template/test/testcases/parser-basic/basic.wgsl.template | Parser fixture input. |
| tools/python/wgsl_template/test/testcases/parser-basic/basic.wgsl.template.pass1 | Parser golden output. |
| tools/python/wgsl_template/test/testcases/parser-comment-removal/test-config.json | Comment-stripping fixture config. |
| tools/python/wgsl_template/test/testcases/parser-comment-removal/comments.wgsl.template | Comment-stripping fixture input. |
| tools/python/wgsl_template/test/testcases/parser-comment-removal/comments.wgsl.template.pass1 | Comment-stripping golden output. |
| tools/python/wgsl_template/test/testcases/parser-empty/test-config.json | Empty-dir parser fixture config. |
| tools/python/wgsl_template/test/testcases/parser-include-basic/test-config.json | Include fixture config. |
| tools/python/wgsl_template/test/testcases/parser-include-basic/main.wgsl.template | Include fixture input. |
| tools/python/wgsl_template/test/testcases/parser-include-basic/main.wgsl.template.pass1 | Include golden output. |
| tools/python/wgsl_template/test/testcases/parser-include-basic/utils.wgsl.template | Included fixture input. |
| tools/python/wgsl_template/test/testcases/parser-include-basic/utils.wgsl.template.pass1 | Included golden output. |
| tools/python/wgsl_template/test/testcases/parser-include-inside-comment/test-config.json | “Include in comments ignored” fixture config. |
| tools/python/wgsl_template/test/testcases/parser-include-inside-comment/main.wgsl.template | Fixture input. |
| tools/python/wgsl_template/test/testcases/parser-include-inside-comment/main.wgsl.template.pass1 | Golden output. |
| tools/python/wgsl_template/test/testcases/parser-include-inside-comment/utils.wgsl.template | Fixture input. |
| tools/python/wgsl_template/test/testcases/parser-include-inside-comment/utils.wgsl.template.pass1 | Golden output. |
| tools/python/wgsl_template/test/testcases/parser-include-invalid-empty/test-config.json | Invalid-include fixture config. |
| tools/python/wgsl_template/test/testcases/parser-include-invalid-empty/main.wgsl.template | Invalid-include fixture input. |
| tools/python/wgsl_template/test/testcases/parser-include-invalid-no-quotes/test-config.json | Invalid-include fixture config. |
| tools/python/wgsl_template/test/testcases/parser-include-invalid-no-quotes/main.wgsl.template | Invalid-include fixture input. |
| tools/python/wgsl_template/test/testcases/parser-include-invalid-nonexist/test-config.json | Missing-include fixture config. |
| tools/python/wgsl_template/test/testcases/parser-include-invalid-nonexist/main.wgsl.template | Missing-include fixture input. |
| tools/python/wgsl_template/test/testcases/parser-include-invalid-path/test-config.json | Invalid-path include fixture config. |
| tools/python/wgsl_template/test/testcases/parser-include-invalid-path/main.wgsl.template | Invalid-path include fixture input. |
| tools/python/wgsl_template/test/testcases/parser-include-invalid-unclosed-quotes/test-config.json | Unclosed-quote include fixture config. |
| tools/python/wgsl_template/test/testcases/parser-include-invalid-unclosed-quotes/main.wgsl.template | Unclosed-quote include fixture input. |
| tools/python/wgsl_template/test/testcases/parser-include-multi-layer/test-config.json | Nested-include fixture config. |
| tools/python/wgsl_template/test/testcases/parser-include-multi-layer/main.wgsl.template | Nested-include fixture input. |
| tools/python/wgsl_template/test/testcases/parser-include-multi-layer/main.wgsl.template.pass1 | Nested-include golden output. |
| tools/python/wgsl_template/test/testcases/parser-include-multi-layer/graphics.wgsl.template | Nested-include fixture input. |
| tools/python/wgsl_template/test/testcases/parser-include-multi-layer/graphics.wgsl.template.pass1 | Nested-include golden output. |
| tools/python/wgsl_template/test/testcases/parser-include-multi-layer/math.wgsl.template | Nested-include fixture input. |
| tools/python/wgsl_template/test/testcases/parser-include-multi-layer/math.wgsl.template.pass1 | Nested-include golden output. |
| tools/python/wgsl_template/test/testcases/parser-include-multi-layer/constants.wgsl.template | Nested-include fixture input. |
| tools/python/wgsl_template/test/testcases/parser-include-multi-layer/constants.wgsl.template.pass1 | Nested-include golden output. |
| tools/python/wgsl_template/test/testcases/parser-include-space-prefix/test-config.json | Leading-space include fixture config. |
| tools/python/wgsl_template/test/testcases/parser-include-space-prefix/main.wgsl.template | Leading-space include fixture input. |
| tools/python/wgsl_template/test/testcases/parser-include-space-prefix/main.wgsl.template.pass1 | Leading-space include golden output. |
| tools/python/wgsl_template/test/testcases/parser-include-space-prefix/constants.wgsl.template | Included fixture input. |
| tools/python/wgsl_template/test/testcases/parser-include-space-prefix/constants.wgsl.template.pass1 | Included golden output. |
| tools/python/wgsl_template/test/testcases/parser-macro-basic/test-config.json | Macro fixture config. |
| tools/python/wgsl_template/test/testcases/parser-macro-basic/basic.wgsl.template | Macro fixture input. |
| tools/python/wgsl_template/test/testcases/parser-macro-basic/basic.wgsl.template.pass1 | Macro golden output. |
| tools/python/wgsl_template/test/testcases/parser-macro-chain/test-config.json | Macro chaining fixture config. |
| tools/python/wgsl_template/test/testcases/parser-macro-chain/chain.wgsl.template | Macro chaining fixture input. |
| tools/python/wgsl_template/test/testcases/parser-macro-chain/chain.wgsl.template.pass1 | Macro chaining golden output. |
| tools/python/wgsl_template/test/testcases/parser-macro-complex-values/test-config.json | Complex macro value fixture config. |
| tools/python/wgsl_template/test/testcases/parser-macro-complex-values/complex.wgsl.template | Complex macro value fixture input. |
| tools/python/wgsl_template/test/testcases/parser-macro-complex-values/complex.wgsl.template.pass1 | Complex macro value golden output. |
| tools/python/wgsl_template/test/testcases/parser-macro-invalid-circular/test-config.json | Circular macro fixture config. |
| tools/python/wgsl_template/test/testcases/parser-macro-invalid-circular/circular.wgsl.template | Circular macro fixture input. |
| tools/python/wgsl_template/test/testcases/parser-macro-invalid-duplicate/test-config.json | Duplicate macro fixture config. |
| tools/python/wgsl_template/test/testcases/parser-macro-invalid-duplicate/duplicate.wgsl.template | Duplicate macro fixture input. |
| tools/python/wgsl_template/test/testcases/parser-macro-invalid-empty-value/test-config.json | Empty macro fixture config. |
| tools/python/wgsl_template/test/testcases/parser-macro-invalid-empty-value/empty.wgsl.template | Empty macro fixture input. |
| tools/python/wgsl_template/test/testcases/parser-macro-invalid-syntax/test-config.json | Invalid macro name fixture config. |
| tools/python/wgsl_template/test/testcases/parser-macro-invalid-syntax/invalid.wgsl.template | Invalid macro name fixture input. |
| tools/python/wgsl_template/test/testcases/parser-macro-invalid-whitespace-value/test-config.json | Whitespace-only macro value fixture config. |
| tools/python/wgsl_template/test/testcases/parser-macro-invalid-whitespace-value/whitespace.wgsl.template | Whitespace-only macro value fixture input. |
| tools/python/wgsl_template/test/testcases/parser-macro-name-edge-cases/test-config.json | Macro name edge-cases fixture config. |
| tools/python/wgsl_template/test/testcases/parser-macro-name-edge-cases/names.wgsl.template | Macro name edge-cases fixture input. |
| tools/python/wgsl_template/test/testcases/parser-macro-name-edge-cases/names.wgsl.template.pass1 | Macro name edge-cases golden output. |
| tools/python/wgsl_template/test/testcases/parser-macro-space-prefix/test-config.json | Leading-space macro fixture config. |
| tools/python/wgsl_template/test/testcases/parser-macro-space-prefix/main.wgsl.template | Leading-space macro fixture input. |
| tools/python/wgsl_template/test/testcases/parser-macro-space-prefix/main.wgsl.template.pass1 | Leading-space macro golden output. |
| tools/python/wgsl_template/test/testcases/parser-subfolder/test-config.json | Subfolder include fixture config. |
| tools/python/wgsl_template/test/testcases/parser-subfolder/main.wgsl.template | Subfolder fixture input. |
| tools/python/wgsl_template/test/testcases/parser-subfolder/main.wgsl.template.pass1 | Subfolder golden output. |
| tools/python/wgsl_template/test/testcases/parser-subfolder/shaders/vertex.wgsl.template | Subfolder fixture input. |
| tools/python/wgsl_template/test/testcases/parser-subfolder/shaders/vertex.wgsl.template.pass1 | Subfolder golden output. |
| tools/python/wgsl_template/test/testcases/parser-subfolder/utils/common.wgsl.template | Subfolder fixture input. |
| tools/python/wgsl_template/test/testcases/parser-subfolder/utils/common.wgsl.template.pass1 | Subfolder golden output. |
| tools/python/wgsl_template/test/testcases/generator-basic/test-config.json | Generator fixture config. |
| tools/python/wgsl_template/test/testcases/generator-basic/test.wgsl.template | Generator fixture input. |
| tools/python/wgsl_template/test/testcases/generator-basic/test.wgsl.template.static-cpp-literal.gen | Generator golden output. |
| tools/python/wgsl_template/test/testcases/generator-preserve-code-ref/test-config.json | Preserve-code-ref generator fixture config. |
| tools/python/wgsl_template/test/testcases/generator-preserve-code-ref/test.wgsl.template | Preserve-code-ref fixture input. |
| tools/python/wgsl_template/test/testcases/generator-preserve-code-ref/test.wgsl.template.static-cpp-literal.gen | Preserve-code-ref golden output. |
| tools/python/wgsl_template/test/testcases/generator-param-basic/test-config.json | Param substitution generator fixture config. |
| tools/python/wgsl_template/test/testcases/generator-param-basic/compute-param.wgsl.template | Param substitution fixture input. |
| tools/python/wgsl_template/test/testcases/generator-param-basic/compute-param.wgsl.template.static-cpp-literal.gen | Param substitution golden output. |
| tools/python/wgsl_template/test/testcases/generator-param-basic/compute-no-param.wgsl.template | Param substitution fixture input. |
| tools/python/wgsl_template/test/testcases/generator-param-basic/compute-no-param.wgsl.template.static-cpp-literal.gen | Param substitution golden output. |
| tools/python/wgsl_template/test/testcases/generator-main/test-config.json | $MAIN generator fixture config. |
| tools/python/wgsl_template/test/testcases/generator-main/main.wgsl.template | $MAIN fixture input. |
| tools/python/wgsl_template/test/testcases/generator-main/main.wgsl.template.static-cpp-literal.gen | $MAIN golden output. |
| tools/python/wgsl_template/test/testcases/generator-main-invalid-unclosed/test-config.json | Unclosed $MAIN fixture config. |
| tools/python/wgsl_template/test/testcases/generator-main-invalid-unclosed/test.wgsl.template | Unclosed $MAIN fixture input. |
| tools/python/wgsl_template/test/testcases/generator-main-invalid-syntax/test-config.json | Invalid $MAIN syntax fixture config. |
| tools/python/wgsl_template/test/testcases/generator-main-invalid-syntax/test.wgsl.template | Invalid $MAIN syntax fixture input. |
| tools/python/wgsl_template/test/testcases/generator-main-invalid-nested/test-config.json | Nested $MAIN fixture config. |
| tools/python/wgsl_template/test/testcases/generator-main-invalid-nested/test.wgsl.template | Nested $MAIN fixture input. |
| tools/python/wgsl_template/test/testcases/generator-main-invalid-multiple/test-config.json | Multiple $MAIN fixture config. |
| tools/python/wgsl_template/test/testcases/generator-main-invalid-multiple/test.wgsl.template | Multiple $MAIN fixture input. |
| tools/python/wgsl_template/test/testcases/generator-example-pad/test-config.json | Complex generator fixture config. |
| tools/python/wgsl_template/test/testcases/generator-example-pad/pad.wgsl.template | Complex generator fixture input. |
| tools/python/wgsl_template/test/testcases/generator-example-pad/pad.wgsl.template.static-cpp-literal.gen | Complex generator golden output. |
| tools/python/wgsl_template/test/testcases/build-basic/test-config.json | Build fixture config. |
| tools/python/wgsl_template/test/testcases/build-basic/src/shader/triangle.wgsl.template | Build fixture input template. |
| tools/python/wgsl_template/test/testcases/build-basic/expected/static-cpp/index.h | Build fixture expected header index. |
| tools/python/wgsl_template/test/testcases/build-basic/expected/static-cpp/index_impl.h | Build fixture expected impl index. |
| tools/python/wgsl_template/test/testcases/build-basic/expected/static-cpp/string_table.h | Build fixture expected string table. |
| tools/python/wgsl_template/test/testcases/build-basic/expected/static-cpp/generated/shader/triangle.h | Build fixture expected per-template header. |
| tools/python/wgsl_template/test/testcases/build-basic/expected/dynamic/templates.js | Legacy expected artifact (not exercised by new harness). |
| tools/python/wgsl_template/test/testcases/build-example-pad/test-config.json | Build fixture config. |
| tools/python/wgsl_template/test/testcases/build-example-pad/src/tensor/pad.wgsl.template | Build fixture input template. |
| tools/python/wgsl_template/test/testcases/build-example-pad/expected/static-cpp-literal/index.h | Build fixture expected header index. |
| tools/python/wgsl_template/test/testcases/build-example-pad/expected/static-cpp-literal/index_impl.h | Build fixture expected impl index. |
| tools/python/wgsl_template/test/testcases/build-example-pad/expected/static-cpp-literal/generated/tensor/pad.h | Build fixture expected per-template header. |
| tools/python/wgsl_template/test/testcases/build-example-pad/expected/dynamic/templates.js | Legacy expected artifact (not exercised by new harness). |
| tools/python/wgsl_template/test/testcases/build-directories/test-config.json | Multi-source-dir build fixture config. |
| tools/python/wgsl_template/test/testcases/build-directories/src/common/utils.wgsl.template | Build fixture input template. |
| tools/python/wgsl_template/test/testcases/build-directories/src/effects/blur.wgsl.template | Build fixture input template. |
| tools/python/wgsl_template/test/testcases/build-directories/src/effects/lighting.wgsl.template | Build fixture input template. |
| tools/python/wgsl_template/test/testcases/build-directories/expected/static-cpp/index.h | Build fixture expected header index. |
| tools/python/wgsl_template/test/testcases/build-directories/expected/static-cpp/index_impl.h | Build fixture expected impl index. |
| tools/python/wgsl_template/test/testcases/build-directories/expected/static-cpp/string_table.h | Build fixture expected string table. |
| tools/python/wgsl_template/test/testcases/build-directories/expected/static-cpp/generated/utils.h | Build fixture expected per-template header. |
| tools/python/wgsl_template/test/testcases/build-directories/expected/static-cpp/generated/@effects/blur.h | Build fixture expected per-template header. |
| tools/python/wgsl_template/test/testcases/build-directories/expected/static-cpp/generated/@effects/lighting.h | Build fixture expected per-template header. |
| tools/ci_build/build_args.py | Remove --wgsl_template CLI option. |
| tools/ci_build/build.py | Remove WGSL template generator CMake flag/vcpkg feature plumbing. |
| cmake/CMakeLists.txt | Remove onnxruntime_WGSL_TEMPLATE option. |
| cmake/onnxruntime_providers_webgpu.cmake | Switch WGSL generation from npm to Python + add ctest hook. |
| cmake/external/onnxruntime_external_deps.cmake | Note dynamic duktape path is now unreachable. |
| cmake/deps.txt | Note duktape planned removal once dynamic generator is deleted. |
| onnxruntime/core/providers/webgpu/wgsl_templates/wgsl_gen.h | Add TODO noting dynamic path now unreachable. |
| onnxruntime/core/providers/webgpu/wgsl_templates/wgsl_gen.cc | Add TODO noting dynamic path now unreachable. |
| onnxruntime/core/providers/webgpu/wgsl_templates/README.md | Update docs to Python generator + remove dynamic-mode guidance. |
| onnxruntime/core/providers/webgpu/wgsl_templates/package.json | Remove Node.js dependency manifest. |
| onnxruntime/core/providers/webgpu/wgsl_templates/package-lock.json | Remove npm lockfile. |
| onnxruntime/core/providers/webgpu/wgsl_templates/.gitignore | Remove node_modules ignore. |
| .github/workflows/windows_webgpu.yml | Remove wgsl_template matrix/flag usage. |
Files not reviewed (1)
- onnxruntime/core/providers/webgpu/wgsl_templates/package-lock.json: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…dor test fixtures
dca9d9c to
549a11a
Compare
Description
Replaces the npm-based
@fs-eire/wgsl-templateNode.js engine used by the WebGPU EP's static build path with an in-tree, stdlib-only Python implementation underwgsl_templates. The dynamic generator path is removed from the build surface - the code logic remains as unreachable and will be removed in a follow-up.Motivation and Context
The WebGPU EP's WGSL template-to-C++ codegen is currently driven by the @fs-eire/wgsl-template npm package, so any build that touches a .wgsl.template file needs Node.js installed and has to pull the package over the network. This npm package is also no longer maintained.