unicode,test: add Qwen3.5 non-backtracking tokenizer handler and regr…#22110

Merged

pwilkin merged 3 commits into

ggml-org:masterfrom

Kabir08:fix/qwen35-tokenizer-stack-overflow

May 14, 2026

Commits on Apr 19, 2026

unicode,test: add Qwen3.5 non-backtracking tokenizer handler and regression tests
Kabir08
committed

Commits on Apr 27, 2026

fix: enhance regex handling for Qwen3.5 tokenizer to include accent marks
Kabir
committed

Commits on May 14, 2026

cont : remove trailing whitespace
aldehir
committed