Fix space sign disagreement

[jmamou]

In BPE tokenizers, all tokens that occur at the beginning of a word, are stored twice as 'token' and ' token'.
Different tokenizers may use different special characters to represent space and we did not take it into account in _get_assistant_to_target_input_ids. Overlap will increase significantly.
The bug has an impact on the target/draft pairs we reported slowdown.

[keyboardAnt]

To address the issue we discussed on Monday regarding counting the size of the intersection between the vocabularies, I propose the following method instead of replacing specific characters like spaces. For any draft token d, decode it into a string D(d), and tokenize this string into the target vocabulary T(D(d)). If T(D(d)) yields a single target token t, then d and t are considered equal. Otherwise, there is no 1:1 mapping.

[jmamou]

To address the issue we discussed on Monday regarding counting the size of the intersection between the vocabularies, I propose the following method instead of replacing specific characters like spaces. For any draft token d, decode it into a string D(d), and tokenize this string into the target vocabulary T(D(d)). If T(D(d)) yields a single target token t, then d and t are considered equal. Otherwise, there is no 1:1 mapping.

Your proposition assumes that if an assistant token can be tokenized either as a single token or as multiple tokens by the target tokenizer, it will be tokenized as a single token.
Can we ensure it?

[keyboardAnt]

We discussed your concern about my proposal over the phone:

Tokenizers might treat the tokens at the beginning of the tokenized string differently. For example, BPE tokenizers may add a BOS token. If the target tokenizer T is a BPE tokenizer, then we might have:

T(s + D(d)) != T(s) + T(D(d))

for some nonempty string s, where + denotes string concatenation.

Therefore, we agreed to move forward with the solution that replaces the space symbol directly.

The only minor issue is the need to rerun the formatting required by Hugging Face. Running make style should suffice.

[keyboardAnt]

LGTM. Please squash-merge.