🚨🚨🚨 `[NLLB Tokenizer]` Fix the prefix tokens 🚨🚨🚨#22313

Merged

ArthurZucker merged 10 commits into

huggingface:mainfrom

ArthurZucker:fix-nllb-tokenizer

Apr 4, 2023

Commits on Mar 22, 2023

fix the prefix tokens
ArthurZucker
committed

Commits on Mar 31, 2023

Commits on Apr 3, 2023

add legacy behaviour

ArthurZucker
and
sgugger
committed
Merge branch 'main' of https://github.com/huggingface/transformers into fix-nllb-tokenizer
ArthurZucker
committed
update disclaimer, linkissue PR and behaviral changes
ArthurZucker
committed

Commits on Apr 4, 2023