tokenizer: Add fastokens support#41741

DarkLight1337 merged 3 commits into

vllm-project:mainvllm-project/vllm:mainfrom

AlonKejzman:akejzman/fastokensAlonKejzman/vllm:akejzman/fastokensCopy head branch name to clipboard