[keivenchang]

Overview

Adds DeepSeek V4.1 grammar construction and malformed-call recovery and preserves V4 grammar output. Shared native/guided argument handling also preserves raw JSON numbers with default Serde scalar compatibility.

Details

• Share DeepSeekDsmlFormat through a private dialect selector, parser-owned marker constants and matching XGrammar argument styles. V4.1 uses the unified-only builder lookup; the family selector remains the only public dialect choice.
• Reuse deferred-error recovery for malformed V4.1 invocations, advance malformed-header scanning incrementally, and verify whole-input, every-split and pre-finish output through both public consumers.
• Keep Serde arbitrary precision opt-in so default Pythonizer/MessagePack/YAML consumers receive native numeric scalars. Native and guided JSON validity, conformance capture and report readers preserve raw argument tokens through shared owners; Muse retains its existing malformed-text recovery.

Example (before -> after)

input:  structural_tag_builder_for_family("deepseek_v41")
before: None
after:  Some(builder), emitting spaced DSML tags and deepseek_v4_1_xml

Before:

After:

Verification

Rebased onto main after PR #501 merged at 00df0c5f6a039dd2d6622efb0216445b8a7ca527.

At e3aa4dceac6f03840203bb280a3fc82e2e0f1ae0 (tree 055bc04c77e4f8dc29cd5a13d8f75f4378b7eda8), the parser suite passed 1,091 tests, the opt-in numeric matrix passed 10, and the full conformance package passed 117. Formatting and both Clippy checks passed. The two default-ignored structural-tag tests were executed separately with XGrammar 0.2.8: 234 Harmony and 138 XML/DSML cases passed. The PR-specific corpus passed 352 acceptance/rejection samples across 54 grammars, including forced cross-dialect rejection and wrong-style negative controls. One pre-existing unrelated GLM optional verifier remains ignored. Python capture-capability tests passed 14 cases.

Final-revision receipts compared 3,345 recorded cases with zero differences. Native and rendered GOLDEN comparisons each checked 3,525 cases and retained 2 nonblocking differences. All three nonempty receipts match this commit and tree; the strict reporter also rejected six missing/empty/stale/severity/mismatch mutations.

Recovery coverage includes malformed bodies, invalid JSON/headers, duplicate parameters, same/subsequent blocks, None/Response/Reasoning entry states, native/guided routing, every UTF-8 split, output before finish, EOF errors, reset/reuse and legacy projection. Numeric coverage includes large integers/fractions, underflow/overflow, embedded closing markers, native Kimi2/Kimi3/Muse/GLM, named/required guided calls, all three policies and immediate trailing text. Capture production and rendering share indexed raw assembly, with sparse/out-of-order call and malformed-value regressions.

Refreshed all 276 existing DeepSeek V4.1 latest cases from qualified unreleased source 0.8.7+source.3c17eddcb31b168332f580c8b61744d66e94816776cfb117d77ebe2e8685013f, with zero parser errors. The captured parser and collector source is public ancestor 3094cb2655dd95b407dbcb34fa66fa61ef81424a; all 261 complete call arguments exactly match raw deltas. The other 801 YAML files, authored inputs and GOLDEN expectations are unchanged. The two retained Kimi K3 GOLDEN warnings remain existing exceptions.

Regenerated the latest report with conformance-publish --root . --output <output> --revision e3aa4dceac6f03840203bb280a3fc82e2e0f1ae0 --candidate-tree 055bc04c77e4f8dc29cd5a13d8f75f4378b7eda8 --cache-dir <cache>, copied it to conformance/CONFORMANCE_v2.html, and verified generated/canonical/HTTP byte equality in real Chromium. The report displays 2,796 cells and 0.8.7; grouped DS4.1 5-malformed_json_then_two_valid_calls shows Matches golden. Rendered HTML stays local. Backend support for the emitted XGrammar style requires separate deployment qualification. Hosted CI run 40795017708 passed on this exact public head and matching mirror: all 22 validation jobs succeeded, with Pages deployment skipped for the PR. DCO, PR-title validation and release preview also passed.

Review scope and follow-up

I’ll address P0/P1 findings in this PR. Let’s handle P2/P3 in follow-up PRs, tracked in DIS-3280. For AI reviewers, report reproducible P0/P1 findings here; record P2/P3 as follow-up work without requesting changes for them on this PR.

Parser release, the dependent Dynamo integration and deployment backend qualification remain follow-up work under that ticket.

Public Dynamo integration is tracked in ai-dynamo/dynamo#17254. Remaining work: release this library change, update Dynamo's dependency, expose grammar availability separately from parser availability, and add Dynamo regression coverage.

Where should the reviewer start?

parsers/v2/src/structural_tag/builders/deepseek_dsml.rs, parsers/v2/src/structural_tag/builders/mod.rs, parsers/v2/tests/deepseek_structural_tag.rs, parsers/v2/src/unified/deepseek_v41.rs, parsers/v2/src/tool_calling/scan.rs

Dependent Dynamo integration

Draft Dynamo #17541 implements the capability/preflight APIs and Dynamo regressions tracked in Dynamo #17254. It remains DO NOT REVIEW until this change is released; the development Git pin must then become a published crates.io version before merge.

[github-actions]

📊 Parser conformance viewer validated — view in CI summary

[luckyq]

Summary: Adds the DeepSeek V4.1 spaced-DSML structural-tag grammar (deepseek_v4_1_xml) via a private V4/V41 dialect enum and makes the V4.1 unified parser recover later valid calls after a malformed invocation; no blocking or non-blocking findings at head 20ee48bff2f4.

Checked (diff read + code trace at 20ee48bff2f4; tests not run locally)

• V4 dialect constants (BLOCK_START / BLOCK_END / INVOKE_START_PREFIX / INVOKE_END / BLOCK_SEPARATOR, BARE_INVOKE_BEGIN via const split_at) reproduce the previous literals exactly → V4 grammar output unchanged; toolchain pin 1.96.1 covers const split_at.
• tool_call_excludes()[1..] ordering consistent across both dialects; generic consumers (build_text_exclusion, reasoning_content) use the full slice.
• JsonSchemaStyle::DeepseekV41Xml serializes as deepseek_v4_1_xml, accepted by upstream XGrammar (structural_tag.py Literal, cpp/structural_tag.cc); no wire→native conversion, so the missing V41 variant in dynamo-structural-tag is unreachable.
• structural_tag_builder_for_family("deepseek_v41") falls through to unified_only_builder.
• Invoke-boundary InvalidParameter { scan_from }: closes at first INVOKE_END, next_scan_start advances on a char boundary, ParameterHeader/ParameterValue flush paths still use malformed_end; no infinite loop or quadratic rescan.
• Scanner deferred-error path (parse_complete_invoke, finish_ordered_into, uncommitted_block): keeps the first error, does not advance next_index, surfaces at finish if unrecovered; guided-JSON rejection test covers the promised-JSON path.
• Legacy deepseek_v4 ToolParser defers only for spaced V4.1 invokes (dsml.rs:428); V4 invokes unchanged.
• 0.8.4 capture diff: provenance metadata for all 276 records + the one intended semantic change (gpt_oss_invalid_tool_body_then_empty_call now emits get_weather).
• Behavioral note, by design and tested: non-EOF malformed V4.1 invocations now error at finish() instead of on the closing push, so text after an unrecovered malformed block streams before the error.

[keivenchang]

Replying to #493 (review): Thanks for checking the V4/V4.1 grammar split and malformed-invocation recovery. Final head e3aa4dce passed the hosted validation suite.