Fix tokenizer special token handling#67

jackzhxng merged 3 commits intomainmeta-pytorch/tokenizers:mainfrom

jz/hf-tokenizer-special-token-handling-retrymeta-pytorch/tokenizers:jz/hf-tokenizer-special-token-handling-retryCopy head branch name to clipboard