Incorrect Document Content in BlenderBot Tokenizer

[chujiezheng]

The BlenderBot tokenizer has been trained to treat spaces like parts of the tokens (a bit like sentencepiece) so a word will be encoded differently whether it is at the beginning of the sentence (without space) or not. However, the examples in BlenderBot Tokenizer (BlenderbotTokenizer) are the same:

transformers/src/transformers/models/blenderbot/tokenization_blenderbot.py

Line 105 in bd469c4

[6950, 1085, 2]

The same issue also occurs in BlenderbotTokenizerFast:

transformers/src/transformers/models/blenderbot/tokenization_blenderbot_fast.py

Line 64 in bd469c4

[6950, 1085, 2]

[sgugger]

Might be of interest to @ArthurZucker

[ArthurZucker]

Okay, this is simply because the model used, BlenderbotTokenizer.from_pretrained("facebook/blenderbot-3B") has the attribute add_prefix_space set to True by default. If you set it to False we have the expected different output.
Let me open a PR to fix this.