feat(template-webpack-plugin): parallelize TASM encode in a shared worker pool#2634
Conversation
…rker pool Move template encoding into a process-wide `tinypool` worker pool so: - Multi-entry builds encode in parallel up to `cpus().length - 1` threads instead of serializing on the main thread. - Watch-mode rebuilds reuse warm workers across compiles. `Buffer.from(uint8Array)` rewraps the postMessage-demoted buffer back to a real `Buffer` so downstream consumers (rspack's `WasmHash`) can still call `.copy()` on it. The worker file URL always resolves to the built `lib/worker/encode.js` so the static pool initializer works both from `src/` (in-tree tests) and `lib/` (installed consumers).
🦋 Changeset detectedLatest commit: d3dfcfe The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
📝 WalkthroughWalkthroughThis PR introduces parallel TASM template encoding via a shared ChangesParallel Template Encoding via Tinypool
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
…rker/encode.js')` Drops the conditional `.ts`/`.js` detection and `--experimental-strip-types` plumbing. Tests now require `lib/` to be built first, but the resolution itself is a single line and works the same in src/ and lib/ contexts via `createRequire(import.meta.url)`.
Merging this PR will not alter performance
Comparing Footnotes
|
React Example#8214 Bundle Size — 236.51KiB (0%).d3dfcfe(current) vs 189be6c main#8210(baseline) Bundle metrics
|
| Current #8214 |
Baseline #8210 |
|
|---|---|---|
0B |
0B |
|
0B |
0B |
|
0% |
0% |
|
0 |
0 |
|
4 |
4 |
|
197 |
197 |
|
80 |
80 |
|
44.87% |
44.87% |
|
2 |
2 |
|
0 |
0 |
Bundle size by type no changes
| Current #8214 |
Baseline #8210 |
|
|---|---|---|
145.76KiB |
145.76KiB |
|
90.75KiB |
90.75KiB |
Bundle analysis report Branch feat/parallel-template-encode Project dashboard
Generated by RelativeCI Documentation Report issue
React External#1328 Bundle Size — 693.04KiB (0%).d3dfcfe(current) vs 189be6c main#1324(baseline) Bundle metrics
|
| Current #1328 |
Baseline #1324 |
|
|---|---|---|
0B |
0B |
|
0B |
0B |
|
0% |
0% |
|
0 |
0 |
|
3 |
3 |
|
17 |
17 |
|
5 |
5 |
|
8.59% |
8.59% |
|
0 |
0 |
|
0 |
0 |
Bundle analysis report Branch feat/parallel-template-encode Project dashboard
Generated by RelativeCI Documentation Report issue
React Example with Element Template#480 Bundle Size — 199.83KiB (0%).d3dfcfe(current) vs 189be6c main#476(baseline) Bundle metrics
Bundle size by type
|
| Current #480 |
Baseline #476 |
|
|---|---|---|
145.76KiB |
145.76KiB |
|
54.08KiB |
54.08KiB |
Bundle analysis report Branch feat/parallel-template-encode Project dashboard
Generated by RelativeCI Documentation Report issue
Web Explorer#9788 Bundle Size — 901.38KiB (0%).d3dfcfe(current) vs 189be6c main#9784(baseline) Bundle metrics
Bundle size by type
|
| Current #9788 |
Baseline #9784 |
|
|---|---|---|
497.1KiB |
497.1KiB |
|
402.06KiB |
402.06KiB |
|
2.22KiB |
2.22KiB |
Bundle analysis report Branch feat/parallel-template-encode Project dashboard
Generated by RelativeCI Documentation Report issue
React MTF Example#1347 Bundle Size — 207.46KiB (0%).d3dfcfe(current) vs 189be6c main#1343(baseline) Bundle metrics
|
| Current #1347 |
Baseline #1343 |
|
|---|---|---|
0B |
0B |
|
0B |
0B |
|
0% |
0% |
|
0 |
0 |
|
3 |
3 |
|
192 |
192 |
|
77 |
77 |
|
44.38% |
44.38% |
|
2 |
2 |
|
0 |
0 |
Bundle size by type no changes
| Current #1347 |
Baseline #1343 |
|
|---|---|---|
111.23KiB |
111.23KiB |
|
96.23KiB |
96.23KiB |
Bundle analysis report Branch feat/parallel-template-encode Project dashboard
Generated by RelativeCI Documentation Report issue
`availableParallelism()` already reflects cgroup CPU limits on container hosts, so subtracting one for the main thread is wasted capacity rather than safety margin. Drops the `-1`. Bumps `engines.node` to `^18.14 || >=19.4` — the version range where the backported `os.availableParallelism()` is actually available.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d3dfcfe6ce
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
🧹 Nitpick comments (2)
packages/webpack/template-webpack-plugin/test/encode-worker-pool.test.ts (1)
11-11: ⚡ Quick winUse
fileURLToPathfor cross-platform compatibility.
new URL(import.meta.url).pathnamecan produce invalid paths on Windows (e.g.,/C:/path/to/file). Import and usefileURLToPathfromnode:urlinstead:🔧 Suggested fix
-import { dirname } from 'node:path'; +import { dirname } from 'node:path'; +import { fileURLToPath } from 'node:url'; -const context = dirname(new URL(import.meta.url).pathname); +const context = dirname(fileURLToPath(import.meta.url));🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/webpack/template-webpack-plugin/test/encode-worker-pool.test.ts` at line 11, Replace the platform-unsafe pathname extraction used to build context: instead of dirname(new URL(import.meta.url).pathname) use fileURLToPath(import.meta.url) and dirname(...) around that; import fileURLToPath from 'node:url' and update the code that defines the context variable (the usage of dirname and import.meta.url) to call fileURLToPath(import.meta.url) so paths are correct on Windows.packages/webpack/template-webpack-plugin/src/LynxEncodePlugin.ts (1)
21-23: 💤 Low valueClarify worker path resolution assumption.
The worker path
require.resolve('../lib/worker/encode.js')assumes the TypeScript source has been compiled tolib/before this code runs. The PR summary states this "enables the static pool initializer to work both fromsrc/(in-tree tests) andlib/(installed consumers)," but actually it requireslib/worker/encode.jsto exist in both cases.Since tests pass, the build likely happens before test execution. Consider clarifying this in a comment or ensuring the README documents the build requirement for running tests.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/webpack/template-webpack-plugin/src/LynxEncodePlugin.ts` around lines 21 - 23, The code currently sets ENCODE_WORKER_PATH via require.resolve('../lib/worker/encode.js') which assumes compiled output exists; update the file to either (a) add a short clarifying comment above ENCODE_WORKER_PATH documenting that tests and consumers require the TS -> lib build step, or (b) implement a robust resolution fallback for ENCODE_WORKER_PATH that first tries require.resolve('../lib/worker/encode.js') and if that fails falls back to resolving the in-tree source (e.g., ../src/worker/encode.ts or ../src/worker/encode.js) using createRequire(import.meta.url); reference the ENCODE_WORKER_PATH constant and the createRequire/import.meta.url call when changing the resolution logic, or update the README to state the build requirement for running tests if you choose the comment-only option.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@packages/webpack/template-webpack-plugin/src/LynxEncodePlugin.ts`:
- Around line 21-23: The code currently sets ENCODE_WORKER_PATH via
require.resolve('../lib/worker/encode.js') which assumes compiled output exists;
update the file to either (a) add a short clarifying comment above
ENCODE_WORKER_PATH documenting that tests and consumers require the TS -> lib
build step, or (b) implement a robust resolution fallback for ENCODE_WORKER_PATH
that first tries require.resolve('../lib/worker/encode.js') and if that fails
falls back to resolving the in-tree source (e.g., ../src/worker/encode.ts or
../src/worker/encode.js) using createRequire(import.meta.url); reference the
ENCODE_WORKER_PATH constant and the createRequire/import.meta.url call when
changing the resolution logic, or update the README to state the build
requirement for running tests if you choose the comment-only option.
In `@packages/webpack/template-webpack-plugin/test/encode-worker-pool.test.ts`:
- Line 11: Replace the platform-unsafe pathname extraction used to build
context: instead of dirname(new URL(import.meta.url).pathname) use
fileURLToPath(import.meta.url) and dirname(...) around that; import
fileURLToPath from 'node:url' and update the code that defines the context
variable (the usage of dirname and import.meta.url) to call
fileURLToPath(import.meta.url) so paths are correct on Windows.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 77007ba3-0e7e-4aa9-82fe-7b82437cf4f9
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (6)
.changeset/parallel-template-encode.mdpackages/webpack/template-webpack-plugin/etc/template-webpack-plugin.api.mdpackages/webpack/template-webpack-plugin/package.jsonpackages/webpack/template-webpack-plugin/src/LynxEncodePlugin.tspackages/webpack/template-webpack-plugin/src/worker/encode.tspackages/webpack/template-webpack-plugin/test/encode-worker-pool.test.ts
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @lynx-js/autolink-codegen@0.1.0 ### Minor Changes - Add the Native Autolink codegen package. ([#2601](#2601)) ## create-lynx-extension@0.1.0 ### Minor Changes - Add the Native Autolink create-extension package. ([#2587](#2587)) ### Patch Changes - Use published package versions for scaffolded autolink codegen dependencies instead of workspace placeholders. ([#2628](#2628)) - Fix npm bin symlink entrypoint detection for the create extension CLI. ([#2623](#2623)) ## @lynx-js/react@0.121.0 ### Minor Changes - Support `React.createElement(type, props, children)` API. ([#2360](#2360)) ```jsx React.createElement("view", { style }, <text>hello</text>); // equivalent to <view style={style}> <text>hello</text> </view>; React.createElement(MyComponent, { style }, <view />); // equivalent to <MyComponent style={style}> <view /> </MyComponent>; ``` ### Patch Changes - Clear transient snapshot child props when removed snapshot subtrees are detached, preventing compiled `$*` child references from retaining deleted list holder or list item subtrees after removal. ([#2590](#2590)) - Add `createPortal` for rendering a subtree into a different ReactLynx element identified by a `NodesRef`. ([#2543](#2543)) ```tsx function App() { const [host, setHost] = useState(null); return ( <view> <view ref={setHost} /> {host && createPortal(<text>hi</text>, host)} </view> ); } ``` - Default `fireEvent` to `bubbles: true` for the TouchEvent family in testing-library to match Lynx runtime semantics, and stop reassigning the read-only `Event.prototype` accessors which threw `TypeError` in strict mode. ([#2532](#2532)) - Set `bundle-url` on lazy bundle border elements. ([#2537](#2537)) - Stop warning when `runWorklet` receives an invalid or missing main-thread function object. Invalid worklet contexts are still ignored, but nullish handler values no longer produce noisy `MainThreadFunction: Invalid function object` console output. ([#2586](#2586)) - Retain main-thread worklet context references before offscreen snapshot elements are materialized, so event, ref, gesture, and spread callbacks stay alive until the DOM update path can attach them. ([#2592](#2592)) - Update the @lynx-js/tasm dependency to 0.0.39 and align React template attribute descriptors with it. ([#2643](#2643)) - Avoid retaining transformed nested worklet contexts after worklet transformation. ([#2591](#2591)) Nested worklets transformed by the worklet runtime now keep their context recovery metadata through a weak reference, preventing cached transformed worklet functions from keeping list-item worklet contexts alive. ## @lynx-js/docs-mcp-server@0.2.3 ### Patch Changes - fix(docs-mcp): recursively crawl and register nested llms.txt resources ([#2317](#2317)) ## @lynx-js/rspeedy@0.14.4 ### Patch Changes - feat(qrcode): support get entry from api exposed from rspeedy.env.entries ([#2551](#2551)) - Updated dependencies \[[`ad1f90f`](ad1f90f)]: - @lynx-js/chunk-loading-webpack-plugin@0.3.4 - @lynx-js/web-rsbuild-server-middleware@0.20.4 - @lynx-js/cache-events-webpack-plugin@0.0.3 ## @lynx-js/lynx-bundle-rslib-config@0.3.3 ### Patch Changes - Update the @lynx-js/tasm dependency to 0.0.39 and align React template attribute descriptors with it. ([#2643](#2643)) ## @lynx-js/qrcode-rsbuild-plugin@0.4.7 ### Patch Changes - feat(qrcode): support get entry from api exposed from rspeedy.env.entries ([#2551](#2551)) ## @lynx-js/react-rsbuild-plugin@0.16.2 ### Patch Changes - Updated dependencies \[[`3e627b3`](3e627b3), [`7b8d63c`](7b8d63c), [`13a0776`](13a0776), [`a973c54`](a973c54), [`353b1b7`](353b1b7)]: - @lynx-js/template-webpack-plugin@0.11.1 - @lynx-js/react-refresh-webpack-plugin@0.3.6 - @lynx-js/react-alias-rsbuild-plugin@0.16.2 - @lynx-js/use-sync-external-store@1.5.0 - @lynx-js/react-webpack-plugin@0.9.2 - @lynx-js/css-extract-webpack-plugin@0.7.1 ## @lynx-js/web-core@0.20.4 ### Patch Changes - Always clone touch event lists when creating cross-thread events so synthetic touch events only carry structured-clone-safe primitive fields. ([#2636](#2636)) - Conditionally pass Card and Component params based on cardType in background thread. ([#2610](#2610)) - Add bidirectional decode worker heartbreak liveness messages. ([#2599](#2599)) - Add web support for the `<frame>` element by mapping it to `<lynx-view>`. ([#2604](#2604)) - Stop redeclaring `fetch` as a chunk-scope binding. Reusing the host ([#2562](#2562)) `window.fetch` from BTS chunks (instead of capturing the no-op stub the chunk wrapper used to install) lets the renderer issue real network requests. - Updated dependencies \[[`c1db603`](c1db603)]: - @lynx-js/web-elements@0.12.2 - @lynx-js/web-worker-rpc@0.20.4 ## @lynx-js/web-elements@0.12.2 ### Patch Changes - fix: xmarkdown create img incorrectly ([#2540](#2540)) ## @lynx-js/chunk-loading-webpack-plugin@0.3.4 ### Patch Changes - Override `__webpack_require__.e` so a single sync-then chunk load (the ([#2597](#2597)) typical lazy bundle case) bypasses `Promise.all`. It will make first screen in main thread can load lazy bundle synchronously when using dynamic import. ## @lynx-js/react-refresh-webpack-plugin@0.3.6 ### Patch Changes - Widen `@lynx-js/react-webpack-plugin` peer range to include `^0.9.0`. ([#2626](#2626)) ## @lynx-js/template-webpack-plugin@0.11.1 ### Patch Changes - feat(web): enable web binary template by default ([#2545](#2545)) The default encoding format for the web platform template has been changed from JSON to Binary. **Benefits for developers:** - **Smaller output size:** Binary templates are more compact than JSON strings, reducing the final bundle size. - **Faster load performance:** Binary templates parse faster than JSON in the runtime, improving the time-to-interactive for web applications. **How to turn off this feature:** If you encounter any issues with the new binary template format, you can revert to the previous JSON format by setting the environment variable `EXPERIMENTAL_USE_WEB_BINARY_TEMPLATE` to `'false'` or `'0'` before running your build commands. For example: `EXPERIMENTAL_USE_WEB_BINARY_TEMPLATE=false rspeedy build` **Upgrade to `@lynx-js/web-core@0.20.2` could support the new output format** See [`@lynx-js/web-core` Changelog](https://lynx-stack.dev/changelog/lynx-js--web-core) - Run TASM template encoding in a shared `tinypool` worker pool so multi-entry builds encode in parallel and watch-mode rebuilds reuse warm workers. ([#2634](#2634)) - Make `LynxTemplatePlugin.getLynxTemplatePluginHooks` a cross-module singleton so duplicate copies of this package (e.g. from npm hoist conflicts) share the same hooks per compilation. ([#2624](#2624)) - Update the @lynx-js/tasm dependency to 0.0.39 and align React template attribute descriptors with it. ([#2643](#2643)) - Updated dependencies \[[`ee79eff`](ee79eff), [`ded4de9`](ded4de9), [`cf01e94`](cf01e94), [`b989c1c`](b989c1c), [`8417e68`](8417e68)]: - @lynx-js/web-core@0.20.4 ## @lynx-js/react-umd@0.121.0 ## create-rspeedy@0.14.4 ## @lynx-js/react-alias-rsbuild-plugin@0.16.2 ## upgrade-rspeedy@0.14.4 ## @lynx-js/web-rsbuild-server-middleware@0.20.4 ## @lynx-js/web-worker-rpc@0.20.4 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Summary
tinypoolworker pool so multi-entry builds encode in parallel (cpus().length - 1threads) instead of serializing on the main thread.LynxEncodePlugin, lazily spawned on first encode.Buffer.from(uint8Array)rewraps the postMessage-demoted buffer back to a realBufferso downstream consumers (rspack'sWasmHash) can still call.copy()on it.lib/worker/encode.js, so the static pool initializer works both fromsrc/(in-tree tests) andlib/(installed consumers).Test plan
pnpm exec rstest runinpackages/webpack/template-webpack-plugin— 346 / 346 pass (including 2 new tests intest/encode-worker-pool.test.ts)pnpm buildinexamples/reactsucceeds with the new pool pathpool.completed += Nfor N-entry builds and that rebuild does not spawn new workersSummary by CodeRabbit
New Features
Chores
^18.14 || >=19.4.