add function call parser for DeepSeek V3

[finger92]

Motivation

support function call for deepseek model issued by #4379

Modifications

add deepseek v3 function call parser

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

validation steps

environment:
H20 * 8

1. start the sglang instance

python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-V3-0324 --tp 8 --port 30000 --host 0.0.0.0 --mem-fraction-static 0.9 --tool-call-parser deepseekv3 --chat-template deepseek.jinja --grammar-backend outlines

2. send request

curl "http://127.0.0.1:30000/v1/chat/completions" \
-H "Content-Type: application/json" \
-d '{
    "messages": [
        {
            "role": "assistant",
             "content": "Act like you are a helpful assistant."
        },
        {
            "role": "user",
            "content": "Hows the weather like in Hangzhou today"
        }
    ],
    "temperature": 0,
    "max_tokens": 200,
    "model": "deepseek-ai/DeepSeek-V3-0324",
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "query_weather",
                "description": "\\nEnter the English name of the specified city to return todays weather query results\\n    :param city:  City name\\n    :return:  Formatted weather information",
                "parameters": {
                    "properties": {
                        "city": {
                            "type": "string",
                            "description": "The city and state, e.g. Beijing"
                        }
                    },
                    "required": [
                        "city"
                    ],
                    "title": "query_weatherArguments",
                    "type": "object"
                }
            }
        }
    ]
}'

3. you will see the output

{"id":"3e8a97c18a324defb39c78148d874c8b","object":"chat.completion","created":1743724834,"model":"/data/DeepSeek-V3-0324","choices":[{"index":0,"message":{"role":"assistant","content":null,"reasoning_content":null,"tool_calls":[{"id":"0","type":"function","function":{"name":"get_weather","arguments":"{\"location\": \"Hangzhou\"}"}}]},"logprobs":null,"finish_reason":"tool_calls","matched_stop":null}],"usage":{"prompt_tokens":76,"total_tokens":98,"completion_tokens":22,"prompt_tokens_details":null}}

deepseek.jinja

{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='', is_first_sp=true, is_last_user=false) %}{%- for message in messages %}{%- if message['role'] == 'system' %}{%- if ns.is_first_sp %}{% set ns.system_prompt = ns.system_prompt + message['content'] %}{% set ns.is_first_sp = false %}{%- else %}{% set ns.system_prompt = ns.system_prompt + '\n\n' + message['content'] %}{%- endif %}{%- endif %}{%- endfor %}{{ bos_token }}{{ ns.system_prompt }}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{%- set ns.is_first = false -%}{%- set ns.is_last_user = true -%}{{'<｜User｜>' + message['content'] + '<｜Assistant｜>'}}{%- endif %}{%- if message['role'] == 'assistant' and tools is defined and tools is not none %}{%- set ns.is_last_user = false -%}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{%- endif %}{%- set ns.is_first = false %}{%- set ns.is_tool = false -%}{%- set ns.is_output_first = true %}{%- for tool in tools %}{%- if not ns.is_first %}{%- if message['content'] is none %}{{'<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>' + tool['function']['name'] + '\n' + '```json' + '\n' ~ tool['function']['parameters'] ~ '\n' + '```' + '<｜tool▁call▁end｜>'}}{%- else %}{{message['content'] + '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>' + tool['function']['name'] + '\n' + '```json' + '\n' ~ tool['function']['parameters'] ~ '\n' + '```' + '<｜tool▁call▁end｜>'}}{%- endif %}{%- set ns.is_first = true -%}{%- else %}{{'\n' + '<｜tool▁call▁begin｜>funtion<｜tool▁sep｜>' + tool['function']['name'] + '\n' + '```json' + '\n' ~ tool['function']['parameters'] ~ '\n' + '```' + '<｜tool▁call▁end｜>'}}{%- endif %}{%- endfor %}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- if message['role'] == 'assistant' and (tools is not defined or tools is none)%}{%- set ns.is_last_user = false -%}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{{content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_last_user = false -%}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_last_user and not ns.is_tool %}{{'<｜Assistant｜>'}}{% endif %}

Attention

1. I had made some modifications to the chat template to support the function calls format in sglang. (see deepseek.jinja)
2. I found that when using default grammar backend (xgrammar), there will be an error:

RuntimeError: [14:59:15] /project/cpp/grammar_matcher.cc:388: Token id 129279:  is regarded as a special token, and cannot be accepted by the GrammarMatcher

and I cant find token with id "129279" in deepseek model's tokenizor.json. but this work fine with "outlines", so I add "--grammar-backend outlines" to launch cmd.
3. use a smaller temperature. I found that the default temperature will cause model output unstable for function calling

[minleminzui]

great work，let me check it，please wait for a moment :-)

[finger92]

great work，let me check it，please wait for a moment :-)

thanks for replying, actually I found some bugs here. I will update this PR later.

[wangxiaoyang-dev]

great work，let me check it，please wait for a moment :-)

thanks for replying, actually I found some bugs here. I will update this PR later.

@finger92 I found the deepseek.jinja and function_call_parser.py has many chinese character, such as <｜tool▁calls▁begin｜>, <｜tool▁sep｜> etc.

[finger92]

great work，let me check it，please wait for a moment :-)

thanks for replying, actually I found some bugs here. I will update this PR later.

@finger92 I found the deepseek.jinja and function_call_parser.py has many chinese character, such as <｜tool▁calls▁begin｜>, <｜tool▁sep｜> etc.

you are right, you can also see them in the tokenizor.json for deepseek model
https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/main/tokenizer.json
I think deepseek team are using these special charactors in the trainings process

[minleminzui]

@finger92 Why did you turn this pr off?

[finger92]

@finger92 Why did you turn this pr off?

I updated the fork repository but accidentally deleted the original branch...
I created a new PR #5224