perf(core): lazy-load web-tree-sitter runtime

[dexhunter]

What this PR does

This changes the web-tree-sitter JavaScript runtime from a static import to a first-use dynamic import inside the existing asynchronous parser initializer. The runtime remains bundled, parser initialization remains deduplicated, and existing regex fallback behavior is preserved when loading fails.

It also adds focused coverage for lazy loading, concurrent initialization, import-failure latching, fallback behavior, emitted-chunk execution with the real parser and WASM assets, and reset recovery.

Why it's needed

Shell parsing is already asynchronous, but the JavaScript runtime was still part of the CLI's static startup closure before any shell command needed it. Deferring that runtime removes 162,719 bytes from the measured startup closure while keeping it available on first parse.

This implementation was found while running autoresearch with Weco, then reduced, tested, and reviewed for submission.

Reviewer Test Plan

How to verify

cd packages/core
npx vitest run src/utils/shellAstParser.test.ts src/utils/shell-ast-parser-lazy.test.ts
npm run build
npm run typecheck
npm run lint

The production CLI bundle evaluator builds the normal esbuild configuration, runs bundled --version and --help smoke checks, verifies that the runtime remains emitted rather than removed or externalized, and measures the static closure containing the CLI entry and default route.

Revision	Static startup closure
Baseline (01d406f1)	16,156,432 bytes
This PR	15,993,713 bytes
Difference	-162,719 bytes (-1.01%)

This measures bytes deferred from startup, not total bundle size or wall-clock startup time.

Evidence (Before & After)

N/A - no user-visible UI change.

Tested on

OS	Status
macOS	Tested
Windows	Not tested
Linux	Tested

Environment (optional)

Node.js 22 with the repository's locked npm dependencies.

Risk & Scope

• Main risk or tradeoff: the first shell parse now pays the dynamic-import cost; parser initialization was already asynchronous and the existing fallback handles load failures.
• Not validated / out of scope: no wall-clock startup improvement is claimed, and the other dependency families listed in [P2] Lazy load heavy optional dependencies / 重型可选依赖延迟加载 #3225 are unchanged.
• Breaking changes / migration notes: none.

Linked Issues

Part of #3225.

[qwen-code-ci-bot]

Thanks for the PR!

Template looks good ✓ — all required headings present, bilingual, test plan included.

Problem: This is an observed optimization with measurable evidence. The PR shows a 162,719-byte (1.01%) reduction in the static startup closure by deferring web-tree-sitter from synchronous load to first-use dynamic import. The esbuild metafile analysis in the test file confirms the runtime is deferred. Not a bug fix — it's a performance improvement with before/after measurements.

Direction: Aligned with the project's performance goals. Part of #3225 (startup optimization). Deferring heavy dependencies from the static startup path is the standard approach for this kind of work. No auth/sandbox/telemetry concerns.

Size: 13 production lines (9 additions, 4 deletions in shellAstParser.ts), 205 test lines in the new test file. Core path (packages/core/src/utils/), well under all thresholds.

Approach: The scope feels right — minimal, focused change. Converts import Parser from 'web-tree-sitter' to import type Parser from 'web-tree-sitter' and adds a dynamic await import('web-tree-sitter') inside the already-async initParser() function. The existing initPromise deduplication and error fallback handle concurrency and failure cases. No simpler path to suggest — this is the textbook lazy-import pattern.

Moving on to code review. 🔍

中文说明

感谢贡献！

模板完整 ✓ — 所有必要标题齐全，双语，包含测试计划。

问题：这是一个有可测量证据的优化。PR 显示通过将 web-tree-sitter 从同步加载推迟到首次使用时动态导入，静态启动闭包减少了 162,719 字节（1.01%）。测试文件中的 esbuild metafile 分析确认了运行时被延迟加载。不是 bug 修复——是带有 before/after 测量的性能改进。

方向：与项目的性能目标一致。属于 #3225（启动优化）的一部分。将重型依赖从静态启动路径中延迟加载是此类工作的标准方法。无 auth/sandbox/telemetry 相关顾虑。

规模：13 行生产代码（shellAstParser.ts 中 9 行新增、4 行删除），新测试文件 205 行。核心路径（packages/core/src/utils/），远低于所有阈值。

方案：范围合理——最小化的聚焦改动。将 import Parser from 'web-tree-sitter' 转为 import type，并在已有的异步 initParser() 函数中添加动态 await import()。现有的 initPromise 去重和错误回退机制处理了并发和失败场景。没有更简单的方案了——这是教科书式的懒导入模式。

进入代码审查 🔍

— Qwen Code · qwen3.7-max

[qwen-code-ci-bot]

Code Review

The diff is clean and minimal. Two changes to shellAstParser.ts:

1. Static import → type-only import: import type Parser from 'web-tree-sitter'
2. Dynamic import inside the existing async IIFE in initParser(): const { default: ParserClass } = await import('web-tree-sitter')

The existing error handling already covers the dynamic import (the catch block on the IIFE marks parserInitFailed = true). The initPromise deduplication ensures concurrent calls only trigger one import. The regex fallback in isShellCommandReadOnlyAST() handles the case where initialization fails. No correctness bugs, no security holes, no regressions found.

The new test file (shell-ast-parser-lazy.test.ts) covers lazy loading, concurrent initialization, import failure latching, and a full esbuild bundle integration test that verifies the runtime is emitted as a deferred chunk rather than in the static closure. Well-written tests.

Reuse check: No new abstractions or utilities introduced. The PR uses the standard dynamic import pattern — nothing to extract or consolidate.

Test Results

All 151 tests pass (148 existing + 3 new):

 ✓ src/utils/shellAstParser.test.ts (148 tests) 86ms
 ✓ src/utils/shell-ast-parser-lazy.test.ts (3 tests) 7892ms

 Test Files  2 passed (2)
      Tests  151 passed (151)

packages/core builds and typechecks cleanly. Lint passes. (Note: packages/web-shell has pre-existing build/typecheck failures unrelated to this PR.)

Real-Scenario Testing (tmux)

Drove the CLI in tmux with a shell command that exercises the parser's permission-check path. Both builds produce identical behavior.

Before (installed qwen 0.19.9, static import)

$ qwen -p 'run: ls -la' --allowedTools 'Bash(ls*)' -y
Warning: running headless with --yolo / approval-mode=yolo and no sandbox. All tool calls (shell, write, edit) auto-execute at this process's privilege level. Enable a sandbox via --sandbox / QWEN_SANDBOX, or set QWEN_CODE_SUPPRESS_YOLO_WARNING=1 to silence this notice.
Listed all files in the project root. The workspace contains the Qwen Code monorepo with `packages/`, `docs/`, `integration-tests/`, and standard Node.js/TypeScript config files.

After (PR bundle, dynamic import)

$ node dist/cli.js -p 'run: ls -la' --allowedTools 'Bash(ls*)' -y
Warning: running headless with --yolo / approval-mode=yolo and no sandbox. All tool calls (shell, write, edit) auto-execute at this process's privilege level. Enable a sandbox via --sandbox / QWEN_SANDBOX, or set QWEN_CODE_SUPPRESS_YOLO_WARNING=1 to silence this notice.
The directory listing shows the project root — a TypeScript monorepo with `packages/`, `docs/`, `integration-tests/`, and standard Node.js/TypeScript config files.

Both builds started, parsed the shell command through the AST parser, determined it was read-only, and executed it successfully. The dynamic import is transparent to the end user.

中文说明

代码审查

Diff 简洁且最小化。shellAstParser.ts 的两处改动：

1. 静态导入 → 仅类型导入：import type Parser from 'web-tree-sitter'
2. 在 initParser() 已有的异步 IIFE 中添加动态导入：const { default: ParserClass } = await import('web-tree-sitter')

现有的错误处理已覆盖动态导入（IIFE 上的 catch 块会设置 parserInitFailed = true）。initPromise 去重机制确保并发调用只触发一次导入。isShellCommandReadOnlyAST() 中的正则回退机制处理了初始化失败的场景。未发现正确性 bug、安全漏洞或回归。

新测试文件（shell-ast-parser-lazy.test.ts）覆盖了延迟加载、并发初始化、导入失败锁定，以及一个完整的 esbuild 打包集成测试（验证运行时作为延迟分块输出而非静态闭包的一部分）。测试写得不错。

复用检查： 未引入新的抽象或工具函数。PR 使用标准动态导入模式——无需提取或整合。

测试结果

全部 151 个测试通过（148 个现有 + 3 个新增）。packages/core 构建和类型检查通过。Lint 通过。（注：packages/web-shell 有与此 PR 无关的预存构建/类型检查失败。）

真实场景测试（tmux）

在 tmux 中通过 CLI 执行了一个会触发解析器权限检查路径的 shell 命令。两个版本行为一致。动态导入对终端用户完全透明。

— Qwen Code · qwen3.7-max

[qwen-code-ci-bot]

This PR is a clean, minimal optimization that does exactly what it says. The change is 13 production lines — convert a static import to a dynamic import inside the already-async parser initializer. It ships with solid test coverage (mock tests for lazy loading and failure modes, plus a full esbuild integration test proving the runtime is deferred from the static closure).

My independent proposal would have been identical: change the import to type-only, add await import('web-tree-sitter') inside the existing IIFE in initParser(). The PR matches that exactly, plus adds thorough tests I wouldn't have thought to include (the esbuild metafile closure analysis is a nice touch).

The before/after tmux test confirms the dynamic import is transparent at runtime — the shell parser initializes correctly in both builds. Unit tests all pass (151/151).

The tradeoff is real and acknowledged: the first shell parse now pays the dynamic import cost. But initParser() was already async, and the existing fallback handles failures. No user-visible behavior change.

Approving. ✅

中文说明

这个 PR 是一个干净、最小化的优化，完全如其所述。改动只有 13 行生产代码——在已有的异步解析器初始化 IIFE 中将静态导入转为动态导入。配套了扎实的测试覆盖（延迟加载和失败模式的 mock 测试，以及一个完整的 esbuild 集成测试证明运行时从静态闭包中被延迟）。

我的独立方案会完全一致：将导入改为仅类型，在 initParser() 已有的 IIFE 中添加 await import('web-tree-sitter')。PR 完全匹配，还加上了我没想到的细致测试（esbuild metafile 闭包分析是个好补充）。

Before/after tmux 测试确认动态导入在运行时完全透明——两个版本的 shell 解析器都正确初始化。所有单元测试通过（151/151）。

取舍是真实且已被承认的：首次 shell 解析会承担动态导入成本。但 initParser() 本来就是异步的，现有的回退机制处理失败场景。无用户可见的行为变化。

通过 ✅

— Qwen Code · qwen3.7-max

[qwen-code-ci-bot]

LGTM, looks ready to ship. ✅

[qwen-code-ci-bot]

No review findings. Downgraded from Approve to Comment: CI failing: web-shell E2E Smoke (ubuntu-latest, Node 22.x), Test (ubuntu-latest, Node 22.x).

— qwen3.7-max via Qwen Code /review

[qwen-code-ci-bot]

Reviewed — no blockers. Suggestions are inline.

[qwen-code-ci-bot]

[Suggestion] Consider adding an eslint no-restricted-imports rule for web-tree-sitter value imports (outside this file). Without it, someone adding import Parser from 'web-tree-sitter' in another module would silently pull the ~163KB runtime back into the synchronous startup closure, undoing this optimization with no CI check to catch it.

— qwen3.7-max via Qwen Code /review

[qwen-code-ci-bot]

[Suggestion] The new test file covers dynamic import failure and deduplication, but does not cover the case where web-tree-sitter imports successfully yet a WASM binary (tree-sitter.wasm or tree-sitter-bash.wasm) fails to load. Both paths share this .catch() handler, but the code leading to it differs. Adding a test that mocks a successful import but a failing loadWasmBinary would give defense-in-depth for the WASM-specific failure path.

— qwen3.7-max via Qwen Code /review

[wenshao]

✅ Maintainer local verification — LGTM (merge reference)

I built and tested this PR locally in an isolated worktree at the PR head (17f241ea0, already merged with main), on macOS (darwin 24.6.0), Node v22.23.1, with a clean npm ci. Everything the PR claims checks out — including the exact byte number.

1. Reviewer test plan — all green

npx vitest run src/utils/shellAstParser.test.ts src/utils/shell-ast-parser-lazy.test.ts
  → Test Files 2 passed (2) · Tests 151 passed (151)   (148 existing regression + 3 new)
npm run build      → Successfully copied files.   (exit 0)
npm run typecheck  → tsc --noEmit, no errors        (exit 0)
npm run lint       → eslint . --ext .ts,.tsx, clean  (exit 0)

The new shell-ast-parser-lazy.test.ts is well-designed: beyond mock-based lazy-load / dedup / import-failure-latch coverage, its third test runs a real esbuild bundle and asserts the runtime + both WASM inputs land in dynamic-import chunks (not the static closure), then loads the emitted bundle and parses real shell commands with the actual parser + WASM.

Guard check (does the test actually protect the behavior?): I overlaid the pre-PR shellAstParser.ts and re-ran the new spec — all 3 tests fail on baseline (runtime eagerly loaded at import; the static import throws under the failure mock; runtime sits in the static closure). So the test passes only with this change. 👍

2. Independent byte-deferral proof — reproduces the exact −162,719 B

I rebuilt the real production esbuild config (DEV=true npm run bundle) on both revisions, changing only shellAstParser.ts, and diffed the static closure of the chunk that owns the shell parser:

Revision	web-tree-sitter runtime	Parser-chunk static closure
Baseline (import Parser …)	inside parser chunk — eager	13,341,405 B
This PR (import type + await import)	own dynamic chunk — deferred	13,178,686 B
Δ		−162,719 B

That is an exact match to the PR's claimed −162,719 bytes (−1.01%). On the production bundle, web-tree-sitter/tree-sitter.js moves out of the parser chunk into a standalone dynamic-import chunk (chunks/tree-sitter-*.js, inStaticClosure=false).

Worth noting for reviewers: the two WASM binaries (tree-sitter.wasm ≈ 255 KB, tree-sitter-bash.wasm ≈ 1.87 MB) were already dynamically imported before this PR — so the PR honestly claims only the ~160 KB runtime JS, not the ~2.1 MB of WASM. No inflated numbers.

3. Scope & risk

• Change is minimal and correct: import Parser → import type Parser (erased at compile, leaves the static import graph) + a first-use await import('web-tree-sitter') inside the already-async, already-deduplicated initParser().
• Failure path preserved: import failure latches parserInitFailed and callers fall back to the regex path (verified by test Where is the config saved? #2 and the guard run).
• Tradeoff is as stated: the first shell parse pays the dynamic-import cost; parser init was already async. No user-visible behavior change.

Verdict: builds clean, full test plan passes, the new test genuinely guards the behavior, and the headline byte saving reproduces to the byte. No concerns from local verification — good to merge from a correctness/perf standpoint.

_{Verified in an isolated worktree; screenshots are faithful renders of the actual local command output.}

🇨🇳 中文版本（点击展开）

✅ 维护者本地验证 —— 通过（合并参考）

我在隔离的 worktree 中基于 PR HEAD（17f241ea0，已与 main 合并）在 macOS（darwin 24.6.0）、Node v22.23.1 上执行了干净的 npm ci 并完成构建与测试。PR 的所有声明均成立 —— 包括精确到字节的体积数字。

1. Reviewer 测试计划 —— 全部通过（见上图 tests）

npx vitest run … shellAstParser.test.ts … shell-ast-parser-lazy.test.ts
  → 2 个测试文件通过 · 151 个用例通过（148 个既有回归 + 3 个新增）
npm run build      → 成功（exit 0）
npm run typecheck  → tsc --noEmit 无报错（exit 0）
npm run lint       → eslint 无问题（exit 0）

新增的 shell-ast-parser-lazy.test.ts 设计到位：除了基于 mock 的懒加载 / 去重 / 导入失败锁存覆盖外，第三个用例会真实执行一次 esbuild 打包，断言运行时与两个 WASM 输入都落在动态导入 chunk（不在静态闭包）中，随后加载产物并用真实解析器 + WASM 解析真实 shell 命令。

守护性验证（测试是否真的保护了该行为？）： 我覆盖回 PR 之前的 shellAstParser.ts 再跑该新用例 —— 3 个用例在基线上全部失败（导入即立即加载运行时；静态 import 在失败 mock 下抛错；运行时位于静态闭包内）。因此该测试只有在本 PR 下才会通过。👍

2. 独立的体积延迟加载验证 —— 精确复现 −162,719 B（见上图 bytes）

我在两个版本上分别用真实生产 esbuild 配置（DEV=true npm run bundle）重新打包，仅改动 shellAstParser.ts，并对拥有 shell parser 的 chunk 的静态闭包做差分：

版本	web-tree-sitter 运行时	Parser chunk 静态闭包
基线（import Parser …）	位于 parser chunk 内 —— 同步加载	13,341,405 B
本 PR（import type + await import）	独立动态 chunk —— 延迟加载	13,178,686 B
Δ		−162,719 B

与 PR 声明的 −162,719 字节（−1.01%） 完全一致。在生产产物中，web-tree-sitter/tree-sitter.js 从 parser chunk 移到了独立的动态导入 chunk（inStaticClosure=false）。

提醒 reviewer：两个 WASM 二进制（tree-sitter.wasm ≈ 255 KB、tree-sitter-bash.wasm ≈ 1.87 MB）在本 PR 之前就已经是动态导入 —— 所以本 PR 只诚实地声明了 ~160 KB 的运行时 JS，并未把 ~2.1 MB 的 WASM 算进去，没有夸大。

3. 范围与风险

• 改动最小且正确：import Parser → import type Parser（编译期擦除，移出静态导入图）+ 在本就异步、本就去重的 initParser() 内首次使用时 await import('web-tree-sitter')。
• 失败路径保留：导入失败会锁存 parserInitFailed，调用方回退到正则路径（由用例 Where is the config saved? #2 与守护性验证证实）。
• 权衡如所述：首次 shell 解析会付出一次动态导入成本；parser 初始化本就是异步的。无用户可见行为变化。

结论： 构建干净、完整测试计划通过、新测试真正守护了该行为、核心体积收益精确复现到字节。本地验证无顾虑，从正确性/性能角度可以合并。