You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
fix for generation that never stops in Llama3-Instruct variants (#1904)
* fix for loading of special tokens in Llama tokenizer
* fix for Llama tokenizer which can have multiple end tokens
* bug fix
* adding some missing tokens to Llama3 tokenizer
* fixed tests and Llama3Tokenizer init.
* now loading correct eos_token config from Hugging Face checkpoint. Using hack for Keras checkpoint because it does not have this info
---------
Co-authored-by: Martin Görner <[email protected]>
0 commit comments