convert : add t5 tokenizer tests

[ggerganov]

Add tokenizer tests + suggest init with UNK tokens

# get tokenizers
python3 convert-hf-to-gguf-update.py <hf_token>

# generate ggml vocab and tests
python3 convert-hf-to-gguf.py models/tokenizers/t5/ --outfile models/ggml-vocab-t5.gguf --vocab-only

# run the tests
make -j tests
./tests/test-tokenizer-0 models/ggml-vocab-t5.gguf

Currently, a few tests that are failing:

src: '!!!!!!'
res: ' !!!!!!'
tok: 3 17065 55 
main : failed test:    '!!!!!!'
main : detokenized to: ' !!!!!!' instead of ' !!!!!!'
main : expected tokens:      3 ' ',     55 '!',  17065 '!!!!!', 
main : got tokens:           3 ' ',  17065 '!!!!!',     55 '!', 

src: ' '
res: '▅'
tok: 2 
main : failed test:    ' '
main : detokenized to: '▅' instead of ''
main : expected tokens: 
main : got tokens:           2 '▅', 

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

[fairydreaming]

@ggerganov These tokenization test failures are caused by differences in tokenization between the transformers T5 "slow" tokenizer (T5Tokenizer) and "fast" tokenizer (T5TokenizerFast). My Unigram tokenizer implementation is compatible with the "slow" tokenizer. It looks like the default implementation returned by AutoTokenizer.from_pretrained(...) in convert-hf-to-gguf-update.py is T5TokenizerFast. I regenerated test inputs with T5Tokenizer by setting use_fast to false: AutoTokenizer.from_pretrained(..., use_fast=False) and they all passed. Try something like this:

@@ -141,7 +147,10 @@ for model in models:
 
     # create the tokenizer
     try:
-        tokenizer = AutoTokenizer.from_pretrained(f"models/tokenizers/{name}")
+        if name == "t5":
+            tokenizer = AutoTokenizer.from_pretrained(f"models/tokenizers/{name}", use_fast=False)
+        else:
+            tokenizer = AutoTokenizer.from_pretrained(f"models/tokenizers/{name}")
     except OSError as e:
         logger.error(f"Error loading tokenizer for model {name}. The model may not exist or is not accessible with the provided token. Error: {e}")
         continue  # Skip to the next model if the tokenizer can't be loaded
@@ -299,7 +309,10 @@ for model in models:
 
     # create the tokenizer
     try:
-        tokenizer = AutoTokenizer.from_pretrained(f"models/tokenizers/{name}")
+        if name == "t5":
+            tokenizer = AutoTokenizer.from_pretrained(f"models/tokenizers/{name}", use_fast=False)
+        else:
+            tokenizer = AutoTokenizer.from_pretrained(f"models/tokenizers/{name}")
     except OSError as e:
         logger.error(f"Failed to load tokenizer for model {name}. Error: {e}")
         continue  # Skip this model and continue with the next one in the loop

[ggerganov]

Thank you - fixed