[BugFix][VoxCPM2]: split multichar Chinese tokens to match training tokenization#2832

linyueqian merged 2 commits into

vllm-project:mainvllm-project/vllm-omni:mainfrom

Sy0307:fix/voxcpm2-chinese-tokenizerSy0307/vllm-omni:fix/voxcpm2-chinese-tokenizerCopy head branch name to clipboard