Add MiniCPM, Deepseek V2 chat template + clean up `llama_chat_apply_template_internal` #8172

ngxson · 2024-06-27T16:49:55Z

Replaces #6236

This PR replaces the tmpl.find(haystack) != std::string::npos pattern with a lambda function tmpl_contains(haystack)

Also add MiniCPM chat template

You are a helpful assistant<用户>Hello<AI>Hi there<用户>Who are you<AI>I am an assistant<用户>Another question<AI>

I have read the contributing guidelines
Self-reported review complexity:
- Low

ngxson · 2024-06-27T17:22:15Z

@fairydreaming I added deepseek lite chat template to this PR. Without chat template, it speaks somewhat english, but now it only speaks chinese:

> hi, who are you
我是DeepSeek Coder，一个由深度求索公司开发的智能助手。我可以帮助您回答问题和提供信息。请问有什么可以帮助您的吗？

> nice
很高兴您觉得还可以！如果您有任何问题或需要帮助的地方，请随时告诉我。

>

fairydreaming

@ngxson I'm not sure if it's a good idea to call it deepseek-lite chat template. From what I see the following models:

all use the same chat template in tokenizer_config.json, so it's better to call it deepseek2. DeepSeek-V2 was first to use it, so I think it's best to refer in comments to simply DeepSeek-V2 instead of DeepSeek-Coder-V2-Lite-Instruct-GGUF like you did.

fairydreaming · 2024-06-27T18:21:12Z

src/llama.cpp

+            }
+        }
+        if (add_ass) {
+            ss << "Assistant: ";


I think you have extra space after "Assistant:" here, this causes the model to speak Chinese.

Extra space removed in 736c494

fairydreaming · 2024-06-27T18:50:23Z

src/llama.cpp

+                ss << trim(message->content);
+            }
+        }
+    } else if (tmpl == "deepseek-lite" || tmpl_contains("'Assistant: ' + message['content'] + eos_token")) {


Change "deepseek-lite" to "deepseek2"

Changed in 736c494

tests/test-chat-template.cpp

fairydreaming · 2024-06-27T19:04:24Z

tests/test-chat-template.cpp

+        "{% for message in messages %}{{'<|' + message['role'] + '|>' + '\n' + message['content'] + '<|end|>\n' }}{% endfor %}{% if add_generation_prompt and messages[-1]['role'] != 'assistant' %}{{- '<|assistant|>\n' -}}{% endif %}",
+        // MiniCPM-3B-OpenHermes-2.5-v2-GGUF
+        u8"{% for message in messages %}{% if message['role'] == 'user' %}{{'<用户>' + message['content'].strip() + '<AI>'}}{% else %}{{message['content'].strip()}}{% endif %}{% endfor %}",
+        // DeepSeek-Coder-V2-Lite-Instruct-GGUF


Replace with DeepSeek-V2

I think you forgot to change this one.

src/llama.cpp

ngxson · 2024-06-27T20:07:15Z

@fairydreaming Thanks for the info. I was confused between deepseek (v1), deepseek-V2 and lite-V2. With the extra space removed, it's now speaking correct language:

> who are you
 I am DeepSeek Coder, an intelligent assistant developed by DeepSeek, a company in China.

> how do you say "happy new year" in chinese
 In Chinese, "happy new year" can be expressed in several ways. Here are some options:

1. 新年快乐 (Xīnnián kuàilè) - This phrase literally means "new year happy." It's the most common way to say "happy new year" in Chinese.

fairydreaming

There's still one "DeepSeek-Coder-V2-Lite-Instruct-GGUF" in the comments, correct it and should be good to go.

fairydreaming · 2024-06-28T06:22:04Z

tests/test-chat-template.cpp

+        "{% for message in messages %}{{'<|' + message['role'] + '|>' + '\n' + message['content'] + '<|end|>\n' }}{% endfor %}{% if add_generation_prompt and messages[-1]['role'] != 'assistant' %}{{- '<|assistant|>\n' -}}{% endif %}",
+        // MiniCPM-3B-OpenHermes-2.5-v2-GGUF
+        u8"{% for message in messages %}{% if message['role'] == 'user' %}{{'<用户>' + message['content'].strip() + '<AI>'}}{% else %}{{message['content'].strip()}}{% endif %}{% endfor %}",
+        // DeepSeek-Coder-V2-Lite-Instruct-GGUF


I think you forgot to change this one.

…emplate_internal` (ggml-org#8172) * tmp_contains * minicpm chat template * add DeepSeek Lite template * change deepseek-lite to deepseek2 * correct code comment * correct code from master branch

ngxson added 2 commits June 27, 2024 18:46

tmp_contains

be26435

minicpm chat template

385ff52

ngxson added the Review Complexity : Low Trivial changes to code that most beginner devs (or those who want a break) can tackle. e.g. UI fix label Jun 27, 2024

ngxson requested a review from ggerganov June 27, 2024 16:49

add DeepSeek Lite template

7d9117a

ngxson changed the title ~~Add MiniCPM chat template + clean up llama_chat_apply_template_internal~~ Add MiniCPM, Deepseek LITE chat template + clean up llama_chat_apply_template_internal Jun 27, 2024

ngxson requested a review from fairydreaming June 27, 2024 17:19

fairydreaming suggested changes Jun 27, 2024

View reviewed changes

fairydreaming reviewed Jun 27, 2024

View reviewed changes

tests/test-chat-template.cpp Show resolved Hide resolved

fairydreaming reviewed Jun 27, 2024

View reviewed changes

tests/test-chat-template.cpp Outdated Show resolved Hide resolved

fairydreaming reviewed Jun 27, 2024

View reviewed changes

src/llama.cpp Outdated Show resolved Hide resolved

github-actions bot added the testing Everything test related label Jun 27, 2024

change deepseek-lite to deepseek2

736c494

ngxson changed the title ~~Add MiniCPM, Deepseek LITE chat template + clean up llama_chat_apply_template_internal~~ Add MiniCPM, Deepseek V2 chat template + clean up llama_chat_apply_template_internal Jun 27, 2024

fairydreaming approved these changes Jun 28, 2024

View reviewed changes

ngxson added 3 commits June 28, 2024 09:07

correct code comment

da86bd0

Merge branch 'master' into xsn/minicpm_template2

ad84e69

correct code from master branch

5ab3182

ngxson merged commit 26a39bb into ggml-org:master Jun 28, 2024

Add MiniCPM, Deepseek V2 chat template + clean up llama_chat_apply_template_internal #8172

Add MiniCPM, Deepseek V2 chat template + clean up llama_chat_apply_template_internal #8172

Uh oh!

Conversation

ngxson commented Jun 27, 2024

Uh oh!

ngxson commented Jun 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fairydreaming left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fairydreaming Jun 27, 2024

Choose a reason for hiding this comment

Uh oh!

ngxson Jun 27, 2024

Choose a reason for hiding this comment

Uh oh!

fairydreaming Jun 27, 2024

Choose a reason for hiding this comment

Uh oh!

ngxson Jun 27, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

fairydreaming Jun 27, 2024

Choose a reason for hiding this comment

Uh oh!

fairydreaming Jun 28, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ngxson commented Jun 27, 2024

Uh oh!

fairydreaming left a comment

Choose a reason for hiding this comment

Uh oh!

fairydreaming Jun 28, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Add MiniCPM, Deepseek V2 chat template + clean up `llama_chat_apply_template_internal` #8172

Add MiniCPM, Deepseek V2 chat template + clean up `llama_chat_apply_template_internal` #8172

ngxson commented Jun 27, 2024 •

edited

Loading

fairydreaming left a comment •

edited

Loading