这个仓库是对 fast_tokenizer 的学习
gcc-10.5
clang-14.0.3
- 测试分词词典
wget https://bj.bcebos.com/paddlenlp/models/transformers/ernie/vocab.txt-
Notifications
You must be signed in to change notification settings - Fork 0
LeoMax-Xiong/leomax-tokenizers
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
对tokenizers进行封装
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published