refactor(metrics): Replace tiktoken with gpt-tokenizer#1245

Closed

yamadashy wants to merge 8 commits intomainfrom

refactor/replace-tiktoken-with-gpt-tokenizer

Commits on Mar 20, 2026

refactor(metrics): Replace tiktoken with gpt-tokenizer for token counting

yamadashy
and
claude
committed
perf(metrics): Share pre-built encoding data across worker threads

yamadashy
and
claude
committed
chore(metrics): Remove tokenizer benchmark scripts and tiktoken devDependency

yamadashy
and
claude
committed
fix(metrics): Fix workerData access for Tinypool and update website references

yamadashy
and
claude
committed
refactor(metrics): Remove pre-built encoding sharing and add encoding validation

yamadashy
and
claude
committed
docs(metrics): Clarify free() is retained for public API compatibility

yamadashy
and
claude
committed
perf(metrics): Use resolveEncodingAsync to load only needed BPE data

yamadashy
and
claude
committed
fix(metrics): Guard against TOCTOU race in async getTokenCounter

yamadashy
and
claude
committed