[stu-cao]

Overview

Closes #530. Follow-up to #479 and #493. The downstream runtime repair this follows is PrimeIntellect-ai/frontend-crates d76e37f.

DeepSeek V4.1 tool arguments are raw XML strings (style: deepseek_v4_1_xml). A malformed closing-control prefix such as </｜ parameter> can remain ordinary content inside an unfinished string, so generation can continue to the token limit without producing a call. This PR reserves the closing-control prefix in V4.1 argument strings at the grammar level. It is a generation restriction, not payload-aware parser recovery.

Affected crate: dynamo-parsers-v2 0.9.10 (parsers/v2).

Details

• JsonSchemaFormat gains an optional excludes: Vec<String> field, serialized only when non-empty. XGrammar's structural-tag wire format already defines it.
• reserve_v41_closing_prefix walks the built format and adds </｜ only to JsonSchemaFormat nodes whose style is deepseek_v4_1_xml.
• StructuralTagBuilder::build_with_options applies the walk after tool-choice policy resolution and before reasoning wrapping, only when the resolved exclude_special_tokens value is true. This covers auto, required, and named tool choice.
• StructuralTagOptions::exclude_special_tokens: false emits no argument exclusion, so a literal </｜ in a string argument remains representable.
• V4 (deepseek_xml), other parser families, and structured-output (response_format) schemas are unchanged. No force-closing of an unfinished but valid prefix is added.

Bug Fix Evidence

1. Minimal Sample Input

Crate: dynamo-parsers-v2 0.9.10, DeepSeek V4.1 builder. Named tool choice reply with:

{"type":"object","properties":{"text":{"type":"string"}},"required":["text"],"additionalProperties":false}

Default exclude_special_tokens. The raw argument text contains a malformed close with two spaces:

hello</｜  parameter> world

inside an unfinished <｜DSML｜ parameter name="text" string="true"> payload. The runnable case is v41/named/guarded=true/malformed-double-space in v41_argument_exclusion_xgrammar:

CONFORMANCE_GRAMMAR_PYTHON=/tmp/st-530-028/bin/python \
  cargo test -p dynamo-parsers-v2 --test structural_tag_xgrammar \
  v41_argument_exclusion_xgrammar -- --ignored

2. Buggy Output (Before)

Without the fix, the named-choice grammar emits the argument schema as:

{"type":"json_schema","json_schema":{"type":"object","properties":{"text":{"type":"string"}},"required":["text"],"additionalProperties":false},"style":"deepseek_v4_1_xml"}

There is no excludes field. XGrammar 0.2.8 accepts hello</｜ parameter> world as string content (accepted: true, complete: true for the enclosing call), so the malformed close never becomes a parameter end. The runtime symptom recorded in #530 is a missing tool call followed by generation to the unchanged 2,048-token limit.

3. Expected/Fixed Output (After)

The emitted schema now includes:

{"type":"json_schema","json_schema":{"type":"object","properties":{"text":{"type":"string"}},"required":["text"],"additionalProperties":false},"style":"deepseek_v4_1_xml","excludes":["</｜"]}

On XGrammar 0.2.8, the malformed close is rejected (accepted: false), while the valid ending </｜DSML｜ parameter> is still accepted (accepted: true, complete: true). Token-boundary cases where the valid terminator is split after </ or after </｜ are also accepted both guarded and unguarded. With exclude_special_tokens: false, no excludes field is emitted and the literal marker remains accepted.

Validation

Command	Result
cargo fmt --all -- --check	exit 0
cargo clippy -p dynamo-parsers-v2 --all-targets -- -D warnings	exit 0
cargo test -p dynamo-parsers-v2	1267 passed, 0 failed, 9 ignored (28 suites)
CONFORMANCE_GRAMMAR_PYTHON=/tmp/st-530-028/bin/python cargo test -p dynamo-parsers-v2 --test structural_tag_xgrammar -- --ignored	6 passed, 0 failed; XGrammar 0.2.8
CONFORMANCE_GRAMMAR_VERSION=0.2.7 CONFORMANCE_GRAMMAR_PYTHON=/tmp/st-530-027/bin/python cargo test -p dynamo-parsers-v2 --test structural_tag_xgrammar -- --ignored	6 passed, 0 failed; XGrammar 0.2.7 compatibility

Tests and what they prove:

• src/structural_tag/tests.rs::v41_argument_exclusion_is_scoped_and_honors_the_opt_out — exact wire shape across V4.1, V4, Qwen3, and GLM; auto/required/named; default/true/false exclude_special_tokens; with and without prompt-opened reasoning; with a structured-output schema. Only V4.1 argument schemas carry excludes; response schemas do not.
• v41_argument_exclusion_xgrammar — real backend, all three tool-choice modes. Malformed closes (</｜ parameter>, </｜ unknown>, </｜ invoke> inside a string) are rejected only when guarded; valid parameter/invoke/calls endings, empty strings, and Unicode strings are accepted; an unfinished valid prefix is accepted but not completed (no force-closing).
• v41_exclusion_token_boundaries_xgrammar — accept_token with a byte vocabulary plus supplied multi-byte tokens. Proves content+terminator in one token, terminator starting a new token, the new terminator-split-after-slash and terminator-split-after-prefix cases (both (true, true) guarded and unguarded), malformed closes split across tokens rejected under guard, literal markers split across tokens accepted only with the opt-out, and hello</｜ accepted as a valid unfinished prefix.
• v41_exclusion_length_bounds_xgrammar — bounded schema (minLength: 2, maxLength: 3). Pins the current 0.2.8 behavior: with excludes active the bounds are dropped (0/1/4-character values accepted), while the unguarded grammar enforces them; Unicode strings in range are accepted.
• v41_argument_exclusion_does_not_leak_xgrammar — V4 argument text containing </｜ parameter> is still accepted, and a V4.1 structured-output response containing </｜ is accepted.
• tests/check_structural_tag_xgrammar.py — adds tokens requests through accept_token and reports backend_version plus excludes_supported from a V4.1-style probe, so the 0.2.7 compatibility job asserts its documented legacy behavior.

Note: the 0.2.8 run prints Ignoring the length constraints of string root_prop_0 (minLength=2, maxLength=3) while the exclusion is active. That is the known limitation described below, not a test failure.

Conformance

Not applicable. This PR does not change parser events; it changes only the structural-tag grammar emitted for generation. conformance/fixtures-unified-v2/families/deepseek_v41 contains no init.structural_tag cases, so no UNIFIED.* IDs, captures, or report columns are added, and conformance/CONFORMANCE_v2.html was not regenerated.

Why this stays a draft

XGrammar 0.2.8 honors excludes, but its converter drops minLength/maxLength on strings while an exclusion is active (mlc-ai/xgrammar#912). The length-bound test above pins that behavior so the gap is visible. Constraint-preserving work is in mlc-ai/xgrammar#901. Per #530, this PR stays a draft until a backend that keeps those bounds is available and the dependency/version policy is agreed. The length-bound test will flag the fix when such a backend lands.

Where should the reviewer start?

• parsers/v2/src/structural_tag/wire.rs — JsonSchemaFormat.excludes and reserve_v41_closing_prefix.
• parsers/v2/src/structural_tag/builders/mod.rs — when the walk is applied.
• parsers/v2/tests/structural_tag_xgrammar.rs — real-backend contracts, including the two token-boundary cases.
• parsers/v2/tests/check_structural_tag_xgrammar.py — token mode and backend capability probe.

Downstream repair

The runtime repair this follows is PrimeIntellect-ai/frontend-crates d76e37f (fix(parsers): reserve V4.1 raw-string closing prefixes). Its evidence — 816/816 Kimi Vendor Verifier cases plus 8/8 tool-choice checks on DeepSeek V4.1 with Dynamo 1.6.1 and vLLM 0.32 — is recorded in #530.

🤖 Generated with Prime Agent (DeepSeek-V4.1-Flash-1017)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.