perf(core): Replace tiktoken WASM with gpt-tokenizer#1343

Closed

yamadashy wants to merge 11 commits intomainfrom

perf/replace-tiktoken-with-gpt-tokenizer

Commits on Mar 28, 2026

perf(core): Replace tiktoken WASM with gpt-tokenizer and simplify metrics pipeline

yamadashy
and
claude
committed
fix(core): Validate token encoding with Zod enum and remove ineffective fallback

yamadashy
and
claude
committed
fix(core): Add special token fallback and fix tokenCounterFactory race condition

yamadashy
and
claude
committed
perf(core): Use disallowedSpecial instead of two-phase try/catch for token counting

yamadashy
and
claude
committed
perf(core): Use fast path without options for token counting

yamadashy
and
claude
committed
perf(core): Avoid V8 deoptimization from complex catch blocks in countTokens

yamadashy
and
claude
committed
fix(core): Add p50k_edit encoding and special token fallback for file metrics

yamadashy
and
claude
committed
fix(core): Use countTokensPlainText for git metrics and strengthen test assertions

yamadashy
and
claude
committed
fix(core): Use countTokensPlainText for output and worker token counting

yamadashy
and
claude
committed
refactor(core): Remove dead calculateMetricsWorker code

yamadashy
and
claude
committed
perf(core): Use fast-path-with-retry for output and git token counting

yamadashy
and
claude
committed