vocab: Support tokenizer for LFM2.5-8B-A1B#23826
Conversation
CISC
left a comment
CISC
left a comment
There was a problem hiding this comment.
Why both in pre_computed_hashes?
CISC
left a comment
CISC
left a comment
There was a problem hiding this comment.
Hmmm, looking more closely the LFM2 pre-tokenizer does not match LFM2.5, it uses a new regex:
'(?i:[sdmt]|ll|ve|re)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]|\\s+(?!\\S)|\\s
|
Thanks for the feedback @CISC .
I tried to place them both in the table, but then re-running
on regex, the difference is minimal, and testing on real-life use cases didn't show any difference. However, we discovered that tool calling doesn't work with this chat template in llama.cpp (works in other frameworks) and currently debugging it. |
One (the original) should stay in
Still, since there is an actual difference it would be prudent to add an |
tdakhran
force-pushed
the
tarek/feat/liquid7-tokenizer
branch
from
964007f to
6f8fa55
Compare
|
@CISC , we reworked the chat template to use a similar regex to lfm2 here https://huggingface.co/LiquidAI/LFM2.5-8B-A1B/discussions/5 . I moved back existing tokenizer to models, hope it looks good to merge! |
That works too I guess. :) |
* origin/master: vocab : support tokenizer for LFM2.5-8B-A1B (ggml-org#23826) graph : ensure DS32 kq_mask_lid is F32 (ggml-org#23864) server: remove obsolete scripts (ggml-org#23870) ci : update macos release to use macos-26 runner (ggml-org#23878) download: add option to skip_download (ggml-org#23059) mtmd: Add DeepSeekOCR 2 Support (ggml-org#20975) CUDA: Check PTX version on host side to guard PDL dispatch (ggml-org#23530) server: bump timeout to 3600s (ggml-org#23842) model : support for DeepseekV32ForCausalLM with generic DeepSeek Sparse Attention (DSA) implementation (ggml-org#23346) llama: use f16 mask for FA to save VRAM (ggml-org#23764) sync : ggml ggml : bump version to 0.13.1 (ggml/1523) ngram-mod : Add missing include (ggml-org#23857) llama: add llm_graph_input_mtp (ggml-org#23643) app : move licences to llama-app (ggml-org#23824) cuda : disables launch_fattn PDL enrollment due to compiler bug (ggml-org#23825) meta : Add missing `buffer` set in allreduce fallback !COMPUTE clear (ggml-org#23480)
* vocab: Support tokenizer for LFM2.5-8B-A1B * Keep liquid6 tokenizer in models
* vocab: Support tokenizer for LFM2.5-8B-A1B * Keep liquid6 tokenizer in models
3 participants
Overview
LFM2.5-8B-A1B shares architecture with LFM2-8B-A1B but comes with a new extended tokenizer.
This PR adds support for it.
GGUFs are uploaded to LiquidAI/LFM2.5-8B-A1B-GGUF
Requirements