Update test to use new tokenizer type API#2692

Merged

ggerganov merged 22 commits intoggml-org:gguffrom

Aug 21, 2023

Commits on Aug 8, 2023

Commits on Aug 15, 2023

Commits on Aug 17, 2023

Merge branch 'gguf' of https://github.com/ggerganov/llama.cpp into gguf
goerch
committed

Commits on Aug 20, 2023

Commits on Aug 21, 2023

Merge branch 'gguf' of https://github.com/goerch/llama.cpp into gguf
goerch
committed
Use token type API in test-tokenizer-1.cpp
goerch
committed