feat: implement async token counter with network resilience and performance optimizations

[jackjackbits]

Summary

• Replace blocking Runtime::new().block_on() pattern with proper async/await to eliminate deadlock risk
• Add global tokenizer caching with concurrent access using DashMap
• Implement token result caching with AHasher for 30% faster lookups
• Add robust network failure handling with exponential backoff retry logic
• Comprehensive test coverage for concurrency and network resilience

Key Performance Improvements

• Eliminated blocking runtime anti-pattern that could cause deadlocks in async contexts
• Lock-free concurrent access with DashMap replacing RwLock
• Faster hash performance with AHasher vs DefaultHasher (~30% improvement)
• Bounded caches prevent memory bloat (10K token cache, 50 tokenizer cache)
• Global tokenizer sharing eliminates redundant downloads/loading

Network Resilience Features

• Exponential backoff retry (3 attempts, 200ms→30s delay cap)
• Smart retry strategy (retry 5xx/network errors, fail fast on 4xx)
• Download validation with JSON structure checking and corruption detection
• Progress reporting for large downloads (>1MB files)
• Enhanced timeouts (60s total, 15s connect) and proper HTTP client config

Test Coverage

• Concurrent tokenizer creation (race condition prevention)
• Cache eviction under load and size management
• Network timeout/failure simulation with real endpoints
• JSON validation for various corruption scenarios
• Performance benchmarking async vs sync implementations

Files Changed

• src/token_counter.rs - Core async implementation with network resilience
• src/agents/context.rs - Updated to use async token counting in critical paths
• src/context_mgmt/ - Added async helpers and summarization functions
• examples/async_token_counter_demo.rs - Performance demonstration
• Cargo.toml - Added dashmap and ahash dependencies

Expected Impact

• No more deadlock risk from nested runtime creation
• Better scalability for multi-threaded token counting
• Resilient to network issues (HuggingFace outages, slow connections)
• Faster repeated operations through intelligent caching
• Production-ready reliability with comprehensive error handling

Testing

All token counter tests pass (18/18). The implementation maintains backward compatibility while adding significant performance and reliability improvements.

[salman1993]

async in fn name might be a typo

[salman1993]

^ same comment - async in fn name might be a typo

[salman1993]

looks good! if we wanted to speed things up more, we can move from using tokenizers to tiktoken-rs crate since we have to estimate token count anyway

tiktoken is faster + fewer dependencies: huggingface/tokenizers#1519

[salman1993]

@jackjackbits here's a PR (built on top of yours) using tiktoken: #3115

BEFORE (tokenizers):

Async TokenCounter:
   Init time: 1.09214125s
   Count time: 996.875µs
   Total tokens: 42
   Cache size: 5

AFTER (tiktoken):

Async TokenCounter:
   Init time: 37.125µs
   Count time: 313.167µs
   Total tokens: 42
   Cache size: 5

[jackjackbits]

@jackjackbits here's a PR (built on top of yours) using tiktoken: #3115

BEFORE (tokenizers):

Async TokenCounter:

   Init time: 1.09214125s

   Count time: 996.875µs

   Total tokens: 42

   Cache size: 5

AFTER (tiktoken):

Async TokenCounter:

   Init time: 37.125µs

   Count time: 313.167µs

   Total tokens: 42

   Cache size: 5

really great.

[salman1993]

i added a small fix to make CI build pass