feat: Simplify tokenization prefix store to single-model architecture

[sagiahrac]

Summary

This PR refactors the tokenization prefix store architecture to support a single model per store instance, eliminating the need for model name management within the store implementations.

Changes Made

• Simplified LRUTokenStore & TrieTokenStore - Removed multi base model support.
• Updated Indexer interface - Removed modelName parameters from AddTokenization and FindLongestContainedTokens methods. Model identity managed at scheduler/indexer level, not storage level
• Updated tests

Configure the base model name in the indexer config
Remove model name parameters from tokenization method calls
The change aligns with the single-model-per-scheduler architecture where each scheduler instance handles one specific base model.

Solves #190
Part of #167

[Copilot]

Pull request overview

This PR refactors the tokenization prefix store architecture to eliminate multi-model support within individual store instances, simplifying the implementation to a single-model-per-store design. This aligns with a broader architectural shift where model identity is managed at the scheduler/indexer level rather than within the storage layer.

• Removed multi-model management from LRUTokenStore and TrieTokenStore implementations
• Updated Indexer interface to remove modelName parameters from AddTokenization and FindLongestContainedTokens methods
• Updated all tests and consumers to remove model name parameters from tokenization calls

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
pkg/tokenization/prefixstore/trie_store.go	Refactored from ContainedTokenStore (multi-model) to TrieTokenStore (single-model); removed model management logic
pkg/tokenization/prefixstore/lru_store.go	Simplified to single-model architecture by removing per-model cache map and using single LRU cache instance
pkg/tokenization/prefixstore/lru_store_test.go	Updated all test cases to remove modelName parameter from function calls
pkg/tokenization/prefixstore/indexer.go	Updated interface signatures to remove modelName parameters
pkg/tokenization/pool_test.go	Updated mock implementations and test expectations to match new interface signatures
pkg/tokenization/pool.go	Updated indexer method calls to remove modelName arguments

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[vMaroon]

I think index might be a better name here.

[sagiahrac]

Considering that the prefix store caches tokenizations, the term index feels too generic. It’s also already used in both kvcache/kvblock/index.go and kvcache/indexer.go. I think we should choose a name that avoids this overloaded notation and better reflects the actual object the cache is representing.

[vMaroon]

/lgtm
/approve