Tokenizer BPE fixes#7530

Merged

jaime-m-p merged 29 commits intoggml-org:masterfrom

jaime-m-p:tokenizer-bpe-fixes

Jun 18, 2024

Commits on May 24, 2024

Update random test: add_bos_token
jaime-m-p
committed
Add BPE models for testing
jaime-m-p
committed

Commits on May 25, 2024

bugfix: custom regex split fails with codepoint 0
jaime-m-p
committed
Refactor llm_tokenizer_bpe: move code to constructor
jaime-m-p
committed
Update random test: add_eos_token
jaime-m-p
committed
Add BPE models for testing
jaime-m-p
committed
Move 'add_special_bos/eos' logic to llm_tokenizer_bpe
jaime-m-p
committed
Fix falcon punctuation regex
jaime-m-p
committed
Better name functions to append token/bos/eos
jaime-m-p
committed
Move tokenizer flags to vocab structure.
jaime-m-p
committed
Allow lstrip for 'added_tokens'
jaime-m-p
committed
Default values for special_add_bos/eos
jaime-m-p
committed
Fix default value for WPM special_add_eos
jaime-m-p
committed

Commits on May 26, 2024

Better variable names
jaime-m-p
committed

Commits on May 29, 2024

Build vocab.special_tokens_cache using vocab token types
jaime-m-p
committed

Commits on Jun 11, 2024

Merge commit '148995e5' into tokenizer-bpe-fixes
jaime-m-p
committed

Commits on Jun 12, 2024

Generalize 'jina-v2' per token attributes
jaime-m-p
committed
Fix merge: 'smaug'
jaime-m-p
committed

Commits on Jun 13, 2024

update brute force random test
jaime-m-p
committed
Fix 'jina-v2' per token attributes
jaime-m-p
committed
Fix unicode whitespaces (deepseek-coder)
jaime-m-p
committed

Commits on Jun 14, 2024

Fix unicode whitespaces (deepseek-llm)
jaime-m-p
committed
Skip missing byte tokens (falcon)
jaime-m-p
committed
Update brute force random test
jaime-m-p
committed
Better unicode data generation
jaime-m-p
committed

Commits on Jun 15, 2024

Merge branch 'master' into tokenizer-bpe-fixes
jaime-m-p
committed
Fix merge: renamed and deleted files
jaime-m-p
committed

Commits on Jun 17, 2024