llama : tokenizer unicode codepoint categories#8606

Open

jaime-m-p wants to merge 30 commits intoggml-org:masterfrom

jaime-m-p:tokenizer-codepoint-categs

Commits on Jul 20, 2024

Update bruteforce test:
jaime-m-p
committed
Store all unicode codepoint categories
jaime-m-p
committed
Reimplement 'codepoint_flags' as 'codepoint_categ'
jaime-m-p
committed
Update unicode data
jaime-m-p
committed
Decode unicode data categories
jaime-m-p
committed
Replace 'codepoint_flags' with 'codepoint_categ'
jaime-m-p
committed

Commits on Jul 25, 2024

Update unicode data: sorted whitespaces
jaime-m-p
committed
Fix codepoint_categ return types
jaime-m-p
committed
Add unicode_data helper functions
jaime-m-p
committed
Reimplement 'collapsed' unicode categories:
jaime-m-p
committed

Commits on Aug 4, 2024

Add more comments
jaime-m-p
committed
Merge commit '978ba3d8' into tokenizer-codepoint-categs
jaime-m-p
committed

Commits on Aug 5, 2024

minor: remove trailing whitespaces and extra semicolons
jaime-m-p
committed
Use GGML_ASSERT and GGML_ABORT
jaime-m-p
committed
Update bruteforce test: fix pyright complaints
jaime-m-p
committed
Update bruteforce test:
jaime-m-p
committed
Binary constants are a C++14 feature
jaime-m-p
committed
Fix copy/paste wrong variable
jaime-m-p
committed
Fix compiler complaints
jaime-m-p
committed

Commits on Aug 7, 2024

Update bruteforce test: fix binary search
jaime-m-p
committed
Unicode data whitespaces as ranges
jaime-m-p
committed
Reimplement unicode_regex_split()
jaime-m-p
committed

Commits on Aug 13, 2024

Remove invalid assert
jaime-m-p
committed
Update codepoint_categ:
jaime-m-p
committed
Reimplement unicode_regex_split():
jaime-m-p
committed
Original regex for 'tekken'
jaime-m-p
committed
Remove unused function
jaime-m-p
committed
Using 32bit wchar_t by default, uint32_t on Windows
jaime-m-p
committed
Fix previous commit
jaime-m-p
committed
Fix compiler complaints
jaime-m-p
committed