GPT-J Will Not Accept Certain Tokens in Prompt

[danforbes]

GPT-J does not like tokenizing certain characters when they appear in a prompt - so far I have only been able to induce this behavior with a ! character, but I haven't performed an exhaustive search.

llm: ./target/release/llm gptj infer -m ~/.ggml-models/gpt4all-j-v1.3-groovy.bin -p "!"
✓ Loaded 285 tensors (3.8 GB) after 1980ms

[2023-05-11T14:36:15Z ERROR llm] Failed to tokenize initial prompt.

[philpax]

Our current tokenizer is built around scores. Perhaps we should use a simpler tokenizer for the models where it's known no score is present for the tokens?

[LLukas22]

Couldn't we use huggingfaces tokenizer? Then we would have parity with nearly every implementation out there 🤔

[philpax]

Yeah maybe, see #35

[danforbes]

@RedBoxing - can you see if this is fixed on your RWKV branch?

[RedBoxing]

no issues at all !