Fix/extend re replacement seq

[saattrupdan]

This PR is an extension of #763, related to extending the re_replacement_seq regex.

The new NorwAI models use a tokenizer that has the token �., which leads to the same error as was described in the previous issue #762.

This PR extends the fix from #763 to deal with this case, as well as adding a unit test to test various tokenizers, and a comment describing why we need the prefix and suffix in the regex.

[saattrupdan]

@rlouf The tests are failing as we need a Hugging Face token to load the NorwAI tokeniser, which is the reason why this PR is needed in the first place. Would it be possible to have a Hugging Face token as a Github secret to deal with this?

[rlouf]

HUGGINGFACE_API_TOKEN should now be available in actions

[saattrupdan]

HUGGINGFACE_API_TOKEN should now be available in actions

That's great, thanks. Can you please get access to the following models with that token?

1. meta-llama/Meta-Llama-3-8B
2. mistralai/Mistral-7B-v0.3
3. google/gemma-2b
4. NorwAI/NorwAI-Mistral-7B-instruct

Those are the culprits of the failing tests.

Also, just to be sure, the HUGGINGFACE_API_TOKEN is an access token and not a token for the Hugging Face inference API, right?

Alternatively, if that's too much of a hassle, I can simply include the failure cases manually, rather than accessing them from "real" tokenisers. Let me know what you think.

[rlouf]

It's probably better to do this "manually" indeed.

[saattrupdan]

@rlouf Changed the test to a "manual" one now, and all tests pass 🙂

[rlouf]

Awesome, thank you!