feat: use tiktoken-rs instead of tokenizers, single global tokenizer

[salman1993]

we estimate the # tokens + lots of recent open source models use tiktoken

BEFORE (tokenizers):

🚀 Async Token Counter Demo
===========================

📊 Performance Comparison
-------------------------
🔴 Synchronous TokenCounter:
   Init time: 1.123163875s
   Count time: 1.423333ms
   Total tokens: 42

🟢 Async TokenCounter:
   Init time: 1.09214125s
   Count time: 996.875µs
   Total tokens: 42
   Cache size: 5

⚡ Cached TokenCounter (2nd run):
   Count time: 3.375µs
   Total tokens: 42
   Cache size: 5

AFTER (tiktoken):

🚀 Async Token Counter Demo
===========================

📊 Performance Comparison
-------------------------
🔴 Synchronous TokenCounter:
   Init time: 1.15345675s
   Count time: 1.199375ms
   Total tokens: 42

🟢 Async TokenCounter:
   Init time: 37.125µs
   Count time: 313.167µs
   Total tokens: 42
   Cache size: 5

⚡ Cached TokenCounter (2nd run):
   Count time: 3.292µs
   Total tokens: 42
   Cache size: 5
   Token result caching: 95x faster on cached text

The difference in performance is significant, especially init (since we just have one tokenizer) but the count time is also 3x faster

📊 Benchmark Results Analysis (done by goose)

Performance Summary (10,000 tokens)

Implementation	Time	Performance
tiktoken-rs (sync)	4.26 ms	⭐ Best Raw Performance
tiktoken-rs (async - cache hit)	3.38 μs	1,257x faster (cached, not real measurement)
Xenova GPT-4o (tokenizers)	8.13 ms	1.9x slower than tiktoken
Xenova Claude (tokenizers)	8.68 ms	2.0x slower than tiktoken

Performance Summary (100,000 tokens)

Implementation	Time	Scaling
tiktoken-rs (sync)	42.4 ms	⭐ Best Raw Performance
tiktoken-rs (async - cache hit)	33.5 μs	1,265x faster (cached, not real measurement)
Xenova GPT-4o (tokenizers)	83.0 ms	2.0x slower than tiktoken

Key Findings

🎯 Raw Tokenization Performance

• tiktoken-rs is ~2x faster than the old tokenizers library
• Performance scales linearly with text length (good!)
• Very consistent performance (low standard deviation)

📈 Scaling Characteristics

• tiktoken-rs: ~4.2 μs per 1,000 tokens (linear scaling)
• tokenizers: ~8.1 μs per 1,000 tokens (linear scaling)
• Both scale linearly, which is ideal

Real-World Implications

✅ What This Means

1. Raw performance improved 2x by switching to tiktoken-rs
2. Memory usage likely reduced (no need to download/store tokenizer files)
3. Cache provides massive speedup for repeated content
4. Consistent performance across different text sizes

⚠️ Cache Considerations

• The async benchmarks show unrealistic performance due to cache hits
• In real usage, you'll see a mix of cache hits and misses
• First-time tokenization will use the sync performance numbers
• Repeated content (like system prompts) will be very fast

🎯 Bottom Line

• Excellent upgrade: 2x faster raw performance + intelligent caching
• Real-world performance: Expect ~4.2ms per 10k tokens for new content
• Cached content: Nearly instant (microseconds)
• Memory efficient: No large tokenizer files to manage

[salman1993]

To see the benchmarking results for previous tokenizers vs tiktoken and also sync/async, download the zip and open report/index.html

benchmark-tokenization.zip

for 100K tokens, here are the mean times:

Xenova--gpt-4o_100000_tokens:  82.216 ms
Xenova--claude-tokenizer_100000_tokens:  91.377 ms
o200k_base_100000_tokens: 42.356 ms
async_o200k_base_100000_tokens: 33.322 µs  # this one is mostly hitting the cache in our toy example, so not a proper measurement

[jamadeo]

What do we expect the error to be when using tiktoken for claude models? I'm guessing it isn't too big a deal since we are mainly using this to suggest when to summarize?

[jamadeo]

Does it not warn for dead code by default?

[salman1993]

not sure why this one wasn't

[salman1993]

What do we expect the error to be when using tiktoken for claude models?

Anthropic doesn't provide us a tokenizer. The previous Xenova-Claude-Tokenizer is also an approximation (see estimate factor here). There is an API endpoint but its easier/faster to use a local tokenizer.