Add DeepSeek-V4 DSML chat template + tolerate truncated invoke close tag (stacked on #1337) - #1760
Closed
nh13 wants to merge 4 commits into
Closed
Add DeepSeek-V4 DSML chat template + tolerate truncated invoke close tag (stacked on #1337)#1760nh13 wants to merge 4 commits into
nh13 wants to merge 4 commits into
Conversation
DeepSeek-V4 (Flash/Pro) emits tool calls in its native DSML format: <｜DSML｜tool_calls> <｜DSML｜invoke name="get_weather"> <｜DSML｜parameter name="city" string="true">Paris</｜DSML｜parameter> </｜DSML｜invoke> ... </｜DSML｜tool_calls> with multiple <｜DSML｜invoke> per block (native parallel calls). string="true" means a literal string value, string="false" a JSON value. Adds mlx_lm/tool_parsers/deepseek_dsml.py (modeled on minimax_m2) and an _infer_tool_parser entry so the official DSML chat template auto-selects it. The start/end markers use the "<｜DSML｜tool_calls" prefix (dropping the trailing ">"): mlx-lm matches markers by token-id sequence and the ">" merges with the following byte on this tokenizer (same class of issue as ml-explore#1335); the parser extracts the invokes/parameters regardless of the leftover ">". Verified on mlx-community/DeepSeek-V4-Flash-2bit-DQ: 0 -> 39/40 (98%) on a jdhodges-style tool suite (the no-tool-template baseline scored 0), 8/8 on parallel multi-tool cases. Adds tests (single, parallel, mixed string/JSON). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The DeepSeek-V4-Flash MLX quants (4/5/6-bit) systematically emit the invoke close tag truncated to </｜DSML｜inv> instead of the canonical </｜DSML｜invoke>, so the strict close in _INVOKE misses the call and it is returned as plain content. Widen the close to </｜DSML｜inv[^>]*>, scoped to the tag boundary.
DeepSeek-V4 ships no Jinja chat template, only the reference Python encoder (encoding_dsv4.py), so there is nothing for chat_template_type to load and tools are never rendered into the prompt. Add a Python apply_chat_template that reuses the existing deepseek_v32 DSML helpers; only the V4-specific tool wording and the tool_calls block tag live here. Byte-exact with the reference encoder for single-turn system+tools+user prompts (chat and thinking modes) and assistant tool-call replay. - Map the server's enable_thinking kwarg onto the reference thinking_mode. - Normalize a top-level tools kwarg onto a system message (synthesizing an empty one if absent) so tools are not dropped when no system message is present. Enable with chat_template_type: "deepseek_dsml" (paired with the deepseek_dsml tool parser).
encode_arguments_to_dsml json.loads() the arguments (the OpenAI JSON string form), but agent loops replay their own prior tool_calls with arguments already parsed as a dict, which crashed multi-turn rendering with "the JSON object must be str, bytes or bytearray, not dict". Re-serialize a dict before encoding.
Member
|
Thanks for the PR but at the moment the number of PRs is way beyond our capacity to review so I'm closing the non-essential ones so we can actually work on this repo. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1337 (@snagnever's
deepseek_dsmlparser) — that should land first; the commits unique to this PR are the two on top. Also depends on DeepSeek-V4 model support (#1189).Adds the encode side plus a robustness fix #1337's parser needs in practice.
1.
chat_templates/deepseek_dsml.py— DSML chat template. DeepSeek-V4 ships no Jinja template (only the reference Pythonencoding_dsv4.py), so there's nothing forchat_template_typeto load and tools are never rendered into the prompt. This is a Pythonapply_chat_templatereusing the existingdeepseek_v32DSML helpers; only the V4-specific tool wording andtool_callsblock tag live here. Byte-exact with the reference encoder for single-turn system+tools+user prompts (chat and thinking modes) and assistant tool-call replay. It maps the server'senable_thinkingkwarg onto the reference'sthinking_mode, and normalizes a top-leveltools=kwarg onto a system message (synthesizing an empty one if absent) so tools aren't silently dropped when no system message is present. Enable withchat_template_type: "deepseek_dsml".2. Lenient invoke close tag. The 4/5/6-bit MLX quants systematically truncate
</｜DSML｜invoke>to</｜DSML｜inv>; the strict_INVOKEclose misses those calls (they come back as content). Widen it to</｜DSML｜inv[^>]*>.Tests: byte-exact chat-template test (skips when the model's
encoding/dir isn't cached) + a close-tag regression test. No model download required.Marker note: #1501 (merged) replaced the token-based state machine with a text-based one, so the trailing-
>marker workaround this branch inherits is no longer needed onmain— the markers can return to full form once this rebases forward past #1189.