perf(core): add pure-ASCII fast path to text token estimation - #6551
Conversation
estimateTextTokens scanned every string char-by-char via charCodeAt to classify ASCII vs non-ASCII code units. For pure-ASCII text (code, English prose - the common case) a single regex scan using V8's optimized string search replaces the JS loop, and the mixed-text path now counts only non-ASCII units, deriving the ASCII count from the length. The token formula is unchanged, so results are byte-identical for every input; verified exhaustively over all 65536 single code units plus 20k randomized mixed strings against the previous implementation. Median of 6 solo benchmark runs over a deterministic mixed fixture set: 51.9ms -> 32.2ms (-38%, 1.61x).
|
Thanks for the PR! Template looks good ✓ — all required sections present including Measured Impact, Risk & Scope, and bilingual Chinese summary. Problem: This is a measured performance improvement, not a theoretical concern. The benchmark evidence (51.9 ms → 32.2 ms, 1.61× speedup on a 10-fixture set × 2000 iterations) demonstrates a real hotspot in Direction: Aligned. Token estimation is on the hot path for every turn and grows with conversation history. A 38% reduction in its cost is meaningful for interactive latency and guardrail checks. The approach — moving ASCII classification to a regex scan that leverages V8's native string search — is a well-understood optimization pattern. Size: Core path ( Approach: The diff is minimal and focused — only the scan strategy changes, the estimation formula is untouched. No drive-by refactors, no scope creep. The new tests pin exactly the right boundaries (U+007F/U+0080 classification, pure-ASCII at various lengths, consistency when mixing, surrogate pair handling). Moving on to code review. 🔍 中文说明感谢贡献！ 模板完整 ✓ — 所有章节齐全，包括性能影响度量、风险与范围以及中文摘要。 问题：这是有基准数据支撑的性能改进（51.9 ms → 32.2 ms，1.61× 加速），而非理论性优化。关联的 issue（#311、#4748、#6506）确认了 token 估算延迟是当前关注点。 方向：对齐。token 估算处于每轮请求的热路径上，38% 的耗时降低对交互延迟和护栏检查有实际意义。将 ASCII 分类移到正则扫描（利用 V8 原生字符串搜索）是成熟的优化模式。 规模：触及核心路径。生产代码 16 行（10 增 + 6 删），测试 31 行。远低于任何阈值。 方案：diff 聚焦且最小化——只改了扫描策略，估算公式未动。无顺手重构、无范围蔓延。新增测试精准覆盖分类边界和一致性。 进入代码审查 🔍 — Qwen Code · qwen3.7-max |
Code ReviewIndependent proposal: To speed up ASCII-heavy text (the common case), I'd add a fast path that checks whether the entire string is ASCII before entering the per-character loop. If pure ASCII, return Comparison: The PR's approach matches this exactly. Correctness: The regex Reuse: Not applicable — this is an in-place optimization of an existing function, no new utilities or parallel code. No blocking issues found. The change is clean, minimal, and correct. Test ResultsUnit tests: 3 files, 64 tests — all pass ✓ TypeCheck: Downstream consumers: Tmux Real-Scenario TestBefore (installed After (PR code via Both return 11 — behavior identical. (The format difference is just the installed build using markdown bold; the underlying token estimation is the same.) Equivalence Spot-CheckDirect function invocation against the PR code: All match expected values from the original formula. 中文说明代码审查独立方案： 对纯 ASCII 文本（常见场景）添加快路径——在逐字符循环之前检查整个字符串是否为 ASCII，若是直接返回 对比： PR 方案与此完全一致。 正确性： 正则 无阻塞性问题。 改动干净、最小且正确。 测试结果
— Qwen Code · qwen3.7-max |
|
This is a clean, well-evidenced performance optimization. The benchmark numbers are real (38% speedup on a deterministic fixture set), the diff is minimal (10 production lines doing exactly one thing), and the equivalence guarantee holds — every input produces the same output as before, verified by 64 unit tests plus a direct spot-check. My independent proposal (regex fast path + derived ASCII count) matched the PR's approach exactly, so no simpler path was missed. The math is straightforward: if there are no non-ASCII characters, The risk the author flags — classification drift between the regex and the loop — is real but well-mitigated by the exhaustive single-code-unit check they ran and the boundary regression tests they added. Approving. ✅ 中文说明这是一个干净且有充分证据的性能优化。基准数据真实（确定性测试集上 38% 加速），diff 最小化（10 行生产代码只做一件事），等价性保证成立——每个输入的输出与之前相同，经 64 个单元测试和直接抽查验证。 我的独立方案（正则快路径 + 推导 ASCII 数量）与 PR 完全一致，未发现更简路径。数学推导直接：无非 ASCII 字符时 作者提到的风险——正则与循环之间的分类漂移——是真实的，但已通过穷举单码元检查和新增边界回归测试充分缓解。 批准。✅ — Qwen Code · qwen3.7-max |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship. ✅
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Reviewed — no blockers. Suggestion-level recommendations are in the Suggestion summary comment below.
Suggestions — commit
|
Maintainer verification — real local build & drive ✅I built the real compiled code for both sides of this PR and exercised it end-to-end (not just the vitest suite). Verdict: behavior is byte-identical, the speedup reproduces, the new tests have teeth — recommend merge. Method
1. Output equivalence — the core claim ("byte-identical for every input")Compared BEFORE vs AFTER over 95,572 inputs → 0 mismatches:
This independently reproduces (and exceeds) the PR's exhaustive-equivalence claim. 2. Test suite + teeth
3. Performance — reproducedMedian of 9 solo processes per side (200 warmup + 2000 iters × 10 fixtures each):
The PR quotes 51.9 → 32.2 ms (1.61× / −38%). My BEFORE median lands on the same 51.9 ms and AFTER is if anything slightly faster — claim reproduced, not overstated. 4. Static checks (on both touched files)
Notes for the record
Recommendation: LGTM — safe to merge. 中文版（点击展开）维护者验证 —— 本地真实构建并驱动运行 ✅我为本 PR 的两侧分别构建了真实的编译产物并做了端到端驱动（不仅仅跑 vitest）。结论：行为逐位一致、性能提升可复现、新增测试确有约束力 —— 建议合并。 方法
1. 输出等价性 —— 核心主张（"任意输入逐字节一致"）对 BEFORE 与 AFTER 在 95,572 个输入上比对 → 0 处不一致：
该结果独立复现并超过了 PR 声称的穷举等价性验证。 2. 测试套件 + 约束力
3. 性能 —— 已复现每侧 9 个独立进程取中位数（各 200 轮预热 + 2000 次迭代 × 10 样本）：
PR 给出 51.9 → 32.2 ms（1.61× / −38%）。我的 BEFORE 中位数正好落在 51.9 ms，AFTER 甚至略快 —— 主张可复现且未夸大。 4. 静态检查（两个改动文件）
备注
结论：LGTM，可安全合并。 Verification: two git worktrees (head |
| if (!NON_ASCII_RE.test(text)) { | ||
| return Math.ceil(text.length / 4); | ||
| } | ||
|
|
||
| let nonAsciiChars = 0; | ||
| for (let i = 0; i < text.length; i++) { |
There was a problem hiding this comment.
[Suggestion] The fast path helps pure ASCII, but mixed strings with a late non-ASCII unit now scan almost the entire text twice: once in NON_ASCII_RE.test(text), then again from index 0 in the loop. RequestTokenizer and PDF token estimation can pass large mostly-ASCII text with an occasional emoji/CJK character, so this regresses a plausible hot path. Reuse the first match position and continue counting after it.
| if (!NON_ASCII_RE.test(text)) { | |
| return Math.ceil(text.length / 4); | |
| } | |
| let nonAsciiChars = 0; | |
| for (let i = 0; i < text.length; i++) { | |
| const firstNonAscii = text.search(NON_ASCII_RE); | |
| if (firstNonAscii === -1) { | |
| return Math.ceil(text.length / 4); | |
| } | |
| let nonAsciiChars = 1; | |
| for (let i = firstNonAscii + 1; i < text.length; i++) { |
— GPT-5 via Qwen Code /review
3 participants
What this PR does
Speeds up the character-based token estimator by 1.61× (median 51.9 ms → 32.2 ms on a deterministic fixture set, −38%). Pure-ASCII text — the common case for code and English prose — is now classified with a single regex scan that uses the engine's optimized string search instead of a per-character loop, and mixed text counts only non-ASCII code units, deriving the ASCII count from the string length. The estimation formula itself is untouched, so every input produces exactly the same token count as before: this was verified exhaustively over all 65,536 single UTF-16 code units and 20,000 randomized mixed strings compared against the previous implementation, plus a pinned sha256 hash over the benchmark outputs and the existing unit suite.
Why it's needed
Token estimation runs on every text part of a request when tokens are counted for the OpenAI- and Anthropic-compatible generators, so its cost is paid per turn and grows with conversation history; the same estimate also backs guardrails such as
sessionTokenLimitand the gating of large extracted documents. Performance of these hot paths is an active concern in this repo — #311 reports general CLI slowness, #4748 tracks fast-path latency work, #6506 optimizes large-paste handling, and #4502 / #3735 added guardrail and auto-compaction features that lean on token estimates. The old loop's per-charactercharCodeAtclassification is O(n) in JS bytecode with a branch per character; the fast path keeps the same O(n) but moves the scan into the engine's native string search, which is substantially cheaper per character.Measured impact
tokenizer_ms(10-fixture set × 2000 iterations)tsxharness callingcalculateTokensover a deterministic 10-fixture set (ASCII code snippet, CJK-only, emoji/Greek mixed, 5,000-char ASCII, 3,000-char EN/CJK mix, 1,000-char prompt fragment, empty/1-char edge cases, 2,000-char Greek) for 2,000 iterations after 200 warmup rounds; each quoted number is the median of 6 runs measured solo on an otherwise idle machine (Node 22, Linux).69e5191a…), an exhaustive check of all 65,536 single code units plus 20,000 seeded random mixed strings found zero mismatches, and the fullrequest-tokenizerunit suite passes (64 tests).Benchmark harness (reproducible)
How this was found
This hotspot was found and optimized with Weco, an evaluation-driven code-optimization agent. The full optimization run (every candidate tried, its measured metric, and the winning diff) is public here:
Weco run: https://dashboard.weco.ai/share/NsmgRvFRbY-d7sFQXn4g2v_fOG12ySZc
The submitted patch is a hand-cleaned re-implementation of the winning candidate: the search's integer-arithmetic and lookup-table variants were dropped because they either alter IEEE-754 rounding on some inputs or add a 64 KB table for ~1% extra gain — the version here keeps the original arithmetic bit-for-bit and takes only the scan-strategy improvement.
Reviewer Test Plan
How to verify
cd packages/core && npx vitest run src/utils/request-tokenizer/→ 3 files, 64 tests pass. This includes four new regression tests pinning the classification boundary (U+007F is ASCII, U+0080 is not), pure-ASCII results at lengths around the fast path (1…4097 chars), consistency when a single non-ASCII character joins long ASCII text (result identical wherever the character sits), and surrogate-pair counting inside mixed text.npx eslintandnpx prettier --checkon both touched files are clean;tsc --noEmit -p packages/corepasses.mainto reproduce the timing delta.Evidence (Before & After)
N/A (no user-visible change; timing numbers are in Measured impact above).
Tested on
Environment (optional)
Unit tests via vitest; benchmark via
npx tsx(Node 22, no sandbox).Risk & Scope
\u0080–\uffff, so lone or paired surrogates classify identically) and the new boundary regression tests.Linked Issues
Related (no auto-close): #311, #4748, #6506.
中文说明
本 PR 做了什么
将基于字符的 token 估算器提速 1.61×（确定性基准集上中位数 51.9 ms → 32.2 ms，−38%）。纯 ASCII 文本（代码和英文文本的常见情形）现在用一次正则扫描来分类，利用引擎优化过的字符串搜索代替逐字符循环；混合文本只统计非 ASCII 码元，ASCII 数量由字符串长度推导。估算公式本身未改动，因此所有输入的 token 数与之前完全一致：已对全部 65,536 个单个 UTF-16 码元及 20,000 条随机混合字符串与旧实现做了穷举比对，并通过基准输出的 sha256 哈希固定值和现有单元测试套件验证。
为什么需要
在为 OpenAI 和 Anthropic 兼容生成器统计 token 时，估算器会作用于请求中的每个文本部分，成本每轮都要支付并随会话历史增长；同样的估算还支撑
sessionTokenLimit等护栏以及大型提取文档的门控。这些热路径的性能是本仓库的活跃议题——#311 报告 CLI 整体变慢，#4748 跟踪快路径延迟优化，#6506 优化大段粘贴处理，#4502 / #3735 增加的护栏与自动压缩功能都依赖 token 估算。旧实现逐字符调用charCodeAt并对每个字符分支；快路径保持同样的 O(n)，但把扫描移入引擎原生的字符串搜索，单字符成本显著更低。实测影响
见上文表格：
tokenizer_ms51.9 ms → 32.2 ms（1.61× / −38%），10 个固定样本 × 2000 次迭代、200 轮预热，每个数字为独占机器上 6 次运行的中位数（Node 22，Linux）。行为一致性通过输出 sha256 哈希、穷举码元比对（零不一致）及 64 个单元测试验证。发现方式
该热点由评估驱动的代码优化代理 Weco 发现并优化，完整运行记录公开于：https://dashboard.weco.ai/share/NsmgRvFRbY-d7sFQXn4g2v_fOG12ySZc 。提交的补丁是对获胜候选的人工净化重写：搜索得到的整数运算与查找表变体被舍弃，因为前者会在部分输入上改变 IEEE-754 舍入行为，后者为约 1% 的额外收益引入 64 KB 表；本版本逐位保留原有算术，只采纳扫描策略上的改进。
审阅者测试计划
cd packages/core && npx vitest run src/utils/request-tokenizer/→ 3 个文件、64 个测试全部通过，其中包含 4 个新增回归测试：固定分类边界（U+007F 属 ASCII、U+0080 不属）、快路径附近各长度的纯 ASCII 结果、单个非 ASCII 字符加入长 ASCII 文本时结果与位置无关、混合文本中代理对按两个码元计数。npx eslint、npx prettier --check均通过；tsc --noEmit -p packages/core通过。main上分别运行上文基准脚本以复现时延差异。风险与范围
\u0080–\uffff内，孤立或成对代理项分类一致）及新增边界回归测试缓解。关联 Issue
相关（不自动关闭）：#311、#4748、#6506。