feat(tree-sitter-bash): add a pure-TypeScript bash parser and an agent-core-v2 bashParser service - #2016
Merged
Merged
Conversation
🦋 Changeset detectedLatest commit: b23241d The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
commit: |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a9ded4966c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| startIndex: node.startIndex, | ||
| endIndex: node.endIndex, | ||
| isNamed: node.isNamed, | ||
| children: node.children.map(snapshot), |
There was a problem hiding this comment.
When callers parse a long but budget-acceptable arithmetic expression, e.g. echo $((1+1+...)) with a few thousand operands and a larger timeoutMs, the parser returns a deeply left-nested binary_expression tree under the default maxNodes; this recursive map(snapshot) then overflows the JS call stack and makes IBashParserService.parse throw RangeError instead of returning hasError/aborted. Since this service is meant to be the wire-safe RPC adapter, the DTO conversion needs the same iterative approach as materialize.
Useful? React with 👍 / 👎.
Add packages/tree-sitter-bash with the SyntaxNode tree model (UTF-16 code-unit offsets, tree-sitter-bash named-node type names), a parse budget (deadline + node cap) with an aborted ParseResult variant, and a placeholder parse() to be replaced by the real lexer/parser.
Cover the permission-analysis grammar subset: lists, pipelines, commands, words, quotes, expansions, command/process substitution, subshells, the full redirect operator set, heredocs and comments, with tree-sitter-bash named-node type names. Long scan loops check the parse deadline via budget.progress() so large literals cannot exhaust the node cap; parse depth is bounded on all recursion chains.
Add compound commands (if/while/until/for/c-style-for/select/case/ function/compound/do_group), test commands with reference-exact extglob/regex right-hand-side rules, a Pratt expression engine for arithmetic expansion shared across arith/c-for/test modes, arrays and subscripts, declaration/unset commands, ansi-c/translated strings and brace expressions. Reserved words are recognized only in statement position. Case-aware balanced scanning keeps command substitutions intact around case items, expression leftovers are kept as ERROR nodes instead of dropped, and recursion depth is bounded per chain (substitution 150, parse 500, lexer scan 1024).
…erf suites Turn the ad-hoc wasm comparison work into permanent infrastructure: a differential helper pinning reference-equivalent and known-difference samples against the real tree-sitter-bash wasm (478 match / 79 known-diff fixtures plus the official v0.25.0 corpus), a three-way consistency check between fixtures, the known-difference registry and the README, deterministic seeded fuzz with tree-integrity assertions, and performance smoke tests. Converges 15 further divergence groups found by systematic probing and documents the rest; the full suite is 853 tests in ~4s.
Wrap the pure @moonshot-ai/tree-sitter-bash package as
IBashParserService (L1, no dependencies): parse(source, options)
returns a wire-safe BashParseResult whose nodes drop the cyclic
parent link so trees can cross the RPC boundary. Budget exhaustion
surfaces as { ok: false, reason: 'aborted' }, malformed input as
hasError — never a throw.
The registration was written against the removed InstantiationType API (#/_base/di/extensions); switch to ScopeActivation.OnDemand from #/_base/di/scope so the package typechecks and the service registers.
Add a fourth icon-rail tab that exercises the App-scope bash parser service over the debug RPC surface: a source textarea with a parse budget (timeoutMs / maxNodes), a dropdown of curated examples adapted from the tree-sitter-bash differential fixtures, and an expandable syntax tree with per-node type, UTF-16 range and leaf text, plus hasError / aborted / node-count badges.
sailist
force-pushed
the
feat/tree-sitter-bash
branch
from
July 28, 2026 06:06
2925c0a to
8c643fc
Compare
A long left-associative chain (e.g. an arithmetic expression with a few thousand operands) parses into a tree thousands of levels deep while still within budget; the recursive DTO conversion then overflowed the JS call stack and made parse throw RangeError, breaking the never-throws contract. Convert the tree with an explicit stack, the same approach as the parser's own materialize.
…view A thousand-operand left-associative chain fills the textarea with a thousand-level binary_expression tree — the shape that once overflowed the DTO conversion. Deeper chains still parse in-process but cannot cross the JSON RPC transport (V8 call-stack limit in serialization), so the example stays within the wire limit.
The rebase onto main merged pnpm-lock.yaml, invalidating the recorded fetchPnpmDeps hash; use the hash CI computed for the merged lockfile.
Merged
7723qqq
pushed a commit
to 7723qqq/kimi-code
that referenced
this pull request
Aug 15, 2026
…t-core-v2 bashParser service (MoonshotAI#2016) * feat(tree-sitter-bash): scaffold pure-TypeScript bash parser package Add packages/tree-sitter-bash with the SyntaxNode tree model (UTF-16 code-unit offsets, tree-sitter-bash named-node type names), a parse budget (deadline + node cap) with an aborted ParseResult variant, and a placeholder parse() to be replaced by the real lexer/parser. * feat(tree-sitter-bash): add lexer and core recursive-descent parser Cover the permission-analysis grammar subset: lists, pipelines, commands, words, quotes, expansions, command/process substitution, subshells, the full redirect operator set, heredocs and comments, with tree-sitter-bash named-node type names. Long scan loops check the parse deadline via budget.progress() so large literals cannot exhaust the node cap; parse depth is bounded on all recursion chains. * feat(tree-sitter-bash): support the full bash grammar Add compound commands (if/while/until/for/c-style-for/select/case/ function/compound/do_group), test commands with reference-exact extglob/regex right-hand-side rules, a Pratt expression engine for arithmetic expansion shared across arith/c-for/test modes, arrays and subscripts, declaration/unset commands, ansi-c/translated strings and brace expressions. Reserved words are recognized only in statement position. Case-aware balanced scanning keeps command substitutions intact around case items, expression leftovers are kept as ERROR nodes instead of dropped, and recursion depth is bounded per chain (substitution 150, parse 500, lexer scan 1024). * test(tree-sitter-bash): add differential fixtures, corpus, fuzz and perf suites Turn the ad-hoc wasm comparison work into permanent infrastructure: a differential helper pinning reference-equivalent and known-difference samples against the real tree-sitter-bash wasm (478 match / 79 known-diff fixtures plus the official v0.25.0 corpus), a three-way consistency check between fixtures, the known-difference registry and the README, deterministic seeded fuzz with tree-integrity assertions, and performance smoke tests. Converges 15 further divergence groups found by systematic probing and documents the rest; the full suite is 853 tests in ~4s. * feat(agent-core-v2): add App-scope bashParser service Wrap the pure @moonshot-ai/tree-sitter-bash package as IBashParserService (L1, no dependencies): parse(source, options) returns a wire-safe BashParseResult whose nodes drop the cyclic parent link so trees can cross the RPC boundary. Budget exhaustion surfaces as { ok: false, reason: 'aborted' }, malformed input as hasError — never a throw. * chore: add changeset for the bash parser service * chore: update pnpmDeps hash after adding tree-sitter-bash package * fix(agent-core-v2): register bashParser service with ScopeActivation The registration was written against the removed InstantiationType API (#/_base/di/extensions); switch to ScopeActivation.OnDemand from #/_base/di/scope so the package typechecks and the service registers. * feat(kimi-inspect): add Bash Parser view Add a fourth icon-rail tab that exercises the App-scope bash parser service over the debug RPC surface: a source textarea with a parse budget (timeoutMs / maxNodes), a dropdown of curated examples adapted from the tree-sitter-bash differential fixtures, and an expandable syntax tree with per-node type, UTF-16 range and leaf text, plus hasError / aborted / node-count badges. * fix(agent-core-v2): snapshot bash syntax trees iteratively A long left-associative chain (e.g. an arithmetic expression with a few thousand operands) parses into a tree thousands of levels deep while still within budget; the recursive DTO conversion then overflowed the JS call stack and made parse throw RangeError, breaking the never-throws contract. Convert the tree with an explicit stack, the same approach as the parser's own materialize. * feat(kimi-inspect): add a deep-arithmetic example to the Bash Parser view A thousand-operand left-associative chain fills the textarea with a thousand-level binary_expression tree — the shape that once overflowed the DTO conversion. Deeper chains still parse in-process but cannot cross the JSON RPC transport (V8 call-stack limit in serialization), so the example stays within the wire limit. * chore: update pnpmDeps hash for the rebased lockfile The rebase onto main merged pnpm-lock.yaml, invalidating the recorded fetchPnpmDeps hash; use the hash CI computed for the merged lockfile.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issue
None — the problem is explained below.
Problem
The Bash tool's permission matching currently treats the entire command string as a single glob subject, so a compound command like
git status && rm -rf /cannot be split and matched per command. The repo had no bash parser at all, and the off-the-shelf option (web-tree-sitter+ wasm) brings wasm asset loading and runtime-specific packaging concerns.What changed
New
packages/tree-sitter-bash— a pure-TypeScript bash parser (zero runtime deps, no wasm).node.text === source.slice(start, end).{ ok: false, reason: 'aborted' }and malformed input returnshasError: true— parsing never throws, so callers can always degrade safely.New
bashParserdomain inpackages/agent-core-v2.IBashParserService:parse(source, options)adapts the package into the DI container and returns a wire-safe DTO (cyclicparentlinks dropped) so results can cross the RPC boundary. No consumers yet — per-command permission matching is the intended follow-up.Repo wiring:
flake.nixworkspace lists, rootAGENTS.mdproject map, and a changeset (internal capability, no user-facing behavior change).This approach fits Kimi Code because it avoids wasm/native loading in the Node runtime, keeps a deterministic failure mode for hostile input, and keeps the parser free of safety policy so permission rules can evolve in the consuming domains.
Checklist
gen-changesetsskill, or this PR needs no changeset.gen-docsskill, or this PR needs no doc update.