vocab: add normalizer.lowercase support to WPM#23899

Merged

ggerganov merged 4 commits into

ggml-org:masterfrom

o7si:wpm-normalizer-lowercase

Jun 1, 2026

Commits on May 29, 2026

vocab : add jina-embeddings-v2-base-zh (whitespace tokenizer)
o7si
committed

Commits on May 30, 2026

vocab : add normalizer.lowercase support to WPM
o7si
committed

Commits on May 31, 2026

Merge remote-tracking branch 'upstream/master' into wpm-normalizer-lowercase
o7si
committed

Commits on Jun 1, 2026

vocab : default normalizer.lowercase to false for whitespace pre-tokenizer
o7si
committed