Skip to content

Fix BPE merge parsing to handle HuggingFace tokenizer.json format#159

Merged
meta-codesync[bot] merged 1 commit intomainfrom
fix-bpe-merge-parsing
Dec 10, 2025
Merged

Fix BPE merge parsing to handle HuggingFace tokenizer.json format#159
meta-codesync[bot] merged 1 commit intomainfrom
fix-bpe-merge-parsing

Commits

Commits on Dec 8, 2025