(Suggestion) Add EOS after each message turn in your future models

[ngxson]

Feature request / 功能建议

Hello, thanks for releasing this model.

I'm Son from Hugging Face and I'm one of the contributor of llama.cpp

I would like to contact you regarding the subject of chat template

Motivation / 动机

Most of your models does not have EOS or EOT (end-of-turn) token after each message turn, for example:

<|system|>\nYou are a helpful assistant<|user|>\nHello<|assistant|>\nHi there<|user|>"

While this still works, most models nowadays add EOS after each turn. For example, here I add <|end|>:

<|system|>\nYou are a helpful assistant<|end|><|user|>\nHello<|end|><|assistant|>\nHi there<|end|>"

This may seem a small change, but can simplify a lot for downstream implementations because we don't need to handle a special case for GLM models.

Your contribution / 您的贡献

This is outside of my reach for making contribution, but I would love to discuss more on this subject. Thank you.

[zRzRzRzRzRzRzR]

I roughly understand what you mean. The EOS token in the training of the GLM model is <|user|>, which indicates the start of the next model's role, rather than a role like <|end|>.
This also exceeds the processing capacity of my position. I would like to confirm whether what you expressed is correct. If so, I will report this situation to the training team.

[ngxson]

Yes, that's what I'm trying to explain.

Having an <|end|> token after each message may sounds wasting (because we already know that the next message starts with <|user|>), but it makes downstream implementations easier because most chat systems expect some sort of dedicated token to mark the end of a role.

Indeed, because you are already using <|user|> and <|assistant|>, it would be awesome if your future models can re-use the same format as phi-3 model from microsoft. The format is:

<|system|>\nYou are a helpful assistant<|end|>\n<|user|>\nHello<|end|>\n<|assistant|>\nHi there<|end|>\n"

Which simply format each message to "<|" + message.role + "|>\n" + message.content + "<|end|>\n"

Here is the jinja template for reference:

{% for message in messages %}{{'<|' + message['role'] + '|>' + '\n' + message['content'] + '<|end|>\n' }}{% endfor %}{% if add_generation_prompt %}{{ '<|assistant|>\n' }}{% endif %}

[zRzRzRzRzRzRzR]

I understand, I have already forwarded this issue to the model team, and I will reply here once they have an update.

[zRzRzRzRzRzRzR]

Our model group colleagues said they would consider this plan for the next generation model!

[ngxson]

That's awesome, thank you!