Removed extra spaces from Mistral instruction template that were causing model to misbehave

[fschuh]

Mistral seems to be very sensitive to any spaces after [/INST] in its instruction template.

Note that the default instruction template provided by Mistral AI (found in the model's tokenizer_config.json) does not contain any spaces before [INST] and after [/INST]:

{{ bos_token }}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token}}{% else %}{{ raise_exception('Only user and assistant roles are supported!') }}{% endif %}{% endfor %}

Unfortunately, that default template doesn't support system messages, but the one in text-generation-webui does.
However, the instruction template in text-generation-webui has some extra spaces in it.
While running some tests with langchain, I found Mistral-7b-instruct responding incorrectly with this instruction template.
Mixtral-8x7B-instruct was similarly affected.

Here's is a prompt that can reproduce the issue (tested with ExLlamav2_HF):

Correct instruction template (with no extra spaces before [INST] and after [/INST]):

Prompt:

[INST] You are a teacher coming up with questions to ask on a quiz.
Given the following document, please generate a question and answer based on that document.

Example Format:
<Begin Document>
...
<End Document>
QUESTION: question here
ANSWER: answer here

These questions should be detailed and be based explicitly on information in the document. Begin!

<Begin Document>
Our new TEK O2 technology makes our four-season waterproof pants even more breathable. It's guaranteed to keep you dry and comfortable – whatever the activity and whatever the weather. Size & Fit: Slightly Fitted through hip and thigh. \n\nWhy We Love It: Our state-of-the-art TEK O2 technology offers the most breathability we've ever tested. Great as ski pants, they're ideal for a variety of outdoor activities year-round.
<End Document> [/INST]

Response (correct):

QUESTION: What is the guarantee of the four-season waterproof pants with TEK O2 technology?
ANSWER: The four-season waterproof pants with TEK O2 technology are guaranteed to keep you dry and comfortable - whatever the activity and whatever the weather.

Incorrect instruction template (with an extra space before [INST] and after [/INST]):

Prompt:

 [INST] You are a teacher coming up with questions to ask on a quiz.
Given the following document, please generate a question and answer based on that document.

Example Format:
<Begin Document>
...
<End Document>
QUESTION: question here
ANSWER: answer here

These questions should be detailed and be based explicitly on information in the document. Begin!

<Begin Document>
Our new TEK O2 technology makes our four-season waterproof pants even more breathable. It's guaranteed to keep you dry and comfortable – whatever the activity and whatever the weather. Size & Fit: Slightly Fitted through hip and thigh. \n\nWhy We Love It: Our state-of-the-art TEK O2 technology offers the most breathability we've ever tested. Great as ski pants, they're ideal for a variety of outdoor activities year-round. 
<End Document> [/INST] 

Response (incorrect):

1. What type of pants is the company's new TEK O2 technology designed for?
Answer: The company's new TEK O2 technology is designed for their four-season waterproof pants.

Note that the incorrect response above does not follow the format in the prompt's instructions.
Removing the extra spaces from the instruction template fixes it.

Checklist:

• I have read the Contributing guidelines.

[BadisG]

Note that the default instruction template provided by Mistral AI (found in the model's tokenizer_config.json) does not contain any spaces before [INST] and after [/INST]:

I thought it did contain space before [INST]

https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1

text = "< s >[INST] What is your favourite condiment? [/INST]"

Unless I'm misrepresenting what < s > means?

[fschuh]

Note that the default instruction template provided by Mistral AI (found in the model's tokenizer_config.json) does not contain any spaces before [INST] and after [/INST]:

I thought it did contain space before [INST]

https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1

text = "< s >[INST] What is your favourite condiment? [/INST]"

Unless I'm misrepresenting what < s > means?

You can add a space before [INST] and it won't break it.
What breaks it is the space after [/INST]. Please refer to my prompt examples above.

For consistency with Mistral's official template though, I've removed both extra spaces on this fix.
Take a closer look at Mistral's official template and you'll see that it doesn't have any spaces before [INST] and after [/INST].

[oobabooga]

Maybe the space should go here then? Like

{{-' ' + message['content'] + '</s>' -}}

[fschuh]

Maybe the space should go here then? Like

{{-' ' + message['content'] + '</s>' -}}

Mistral's official template doesn't add extra spaces for the assistant messages:

{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token}}

For consistency, I'd keep it as is, which is like Mistral's default.

[oobabooga]

Yes, that's correct. But there seems to be a space after the </s> at least:

{{ bos_token }}
{% for message in messages %}
    {% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}
        {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
    {% endif %}
    
    {% if message['role'] == 'user' %}
        {{ '[INST] ' + message['content'] + ' [/INST]' }}
    {% elif message['role'] == 'assistant' %}
        {{ message['content'] + eos_token + ' ' }}
    {% else %}
        {{ raise_exception('Only user and assistant roles are supported!') }}
    {% endif %}
{% endfor %}

https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1/blob/main/tokenizer_config.json#L32

[oobabooga]

Maybe not for the Mixtral one (see below). It's kind of frustrating that not even Mistral seems to know what the Mistral prompt should be.

{{ bos_token }}
{% for message in messages %}
    {% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}
        {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
    {% endif %}
    
    {% if message['role'] == 'user' %}
        {{ '[INST] ' + message['content'] + ' [/INST]' }}
    {% elif message['role'] == 'assistant' %}
        {{ message['content'] + eos_token }}
    {% else %}
        {{ raise_exception('Only user and assistant roles are supported!') }}
    {% endif %}
{% endfor %}

https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1/blob/main/tokenizer_config.json#L42