Add tekken to the tokenize tool (#119)#124

facebook-github-bot merged 1 commit intomainmeta-pytorch/tokenizers:mainfrom

jz/tokenizer-toolmeta-pytorch/tokenizers:jz/tokenizer-toolCopy head branch name to clipboard