feat: add DeepSeek V4 Flash 0731 renderer

[hallerite]

Summary

• add a hand-coded DeepSeek V4 renderer matching the official Python encoding_dsv4.py contract
• support chat and thinking modes, reasoning effort, DSML tool schemas and calls, ordered tool results, parsing, attribution masks, and safe bridging
• register deepseek-ai/DeepSeek-V4-Flash-0731 under the typed deepseek-v4 configuration and document the new renderer
• generalize the shared test oracle so Jinja checkpoints use apply_chat_template while DeepSeek V4 uses an independent mirror of its official Python encoder
• add DeepSeek V4 to the shared render, attribution, parsing, preservation, bridge, round-trip, and config-parity matrices
• encode contiguous prompt text in one BPE pass; the generalized barrage caught token-ID drift at renderer chunk boundaries even when decoded strings matched
• preserve the official encoder's task-transition, user/tool merge, and strict DSML parsing edge behavior with explicit parity regressions
• merge the latest origin/main (32e6622) with a signed merge commit after re-signing the full PR stack

Validation

• all 4 official DeepSeek encoding fixtures match byte-for-byte
• DeepSeek V4 shared barrage: 68 passed, 1 expected skip
• DeepSeek-focused bridge, round-trip, dedicated, and config matrix: 71 passed, 1 expected skip
• exact task/DSML parity regressions: 19 passed
• official JSON non-object argument exception parity: 4 passed
• focused DeepSeek V4 file after final parity regression: 23 passed
• post-refactor parser and round-trip verification: 36 passed
• full merged-tree suite: 3821 passed, 181 skipped, 1 expected xfail
• repository-wide Ruff lint and format checks pass
• CI-equivalent renderer type check completes without the parser panic exposed during development

Official encoder: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731/blob/main/encoding/encoding_dsv4.py

Note

Add DeepSeekV4Renderer for DeepSeek V4 Flash 0731

• Adds DeepSeekV4Renderer and DeepSeekV4RendererConfig to support the deepseek-ai/DeepSeek-V4-Flash-0731 model, which uses DSML wire format instead of a tokenizer chat template.
• Implements DSML tool rendering, tool-result merging and ordering, thinking retention logic, task-token emission, and response-format embedding in deepseek_v4.py.
• Adds parse_deepseek_v4 to decode model outputs into content, reasoning, and structured DSML tool calls in parsing.py.
• Introduces a model-aware reference oracle in reference_rendering.py to replace tokenizer.apply_chat_template in test parity checks.
• Behavioral Change: DeepSeekV4RendererConfig defaults to enable_thinking=False, drop_thinking=True, and reasoning_effort='low'.

Changes since #140 opened

• Introduced _QUERY_ROLES frozenset constant containing 'user' and 'developer', added _is_query_message helper function, and updated renderers.deepseek_v4.DeepSeekV4Renderer.render_ids to replace hardcoded role checks with _QUERY_ROLES when computing the last query boundary index, filtering historical developer and search-agent messages during thinking drops, determining reasoning retention for assistant messages, and opening new reasoning segments [98cb612]
• Updated renderers.deepseek_v4.DeepSeekV4Renderer.bridge_to_next_turn to pass is_user_query=_is_query_message predicate to should_rerender_for_thinking_retention method [98cb612]
• Added test cases test_bridge_declines_at_developer_query_boundary_when_dropping_thinking and test_bridge_extends_developer_query_when_preserving_all_thinking in tests.test_deepseek_v4 module [98cb612]

^{Macroscope summarized a7217e5.}

---

Note

Medium Risk
New ~860-line renderer on the critical prompt/tokenization path with complex thinking, tool, and bridge semantics; mistakes would skew training/inference token streams for V4, though coverage is extensive and defaults match the official encoder.

Overview
Adds first-class support for deepseek-ai/DeepSeek-V4-Flash-0731, which has no Jinja chat template and instead follows the model repo’s Python encoding_dsv4.py contract.

DeepSeekV4Renderer and DeepSeekV4RendererConfig implement that encoder in the renderer protocol: chat vs thinking mode, optional reasoning-effort prefixes, historical reasoning drop (disabled when tools are present), DSML tool schemas/calls, merged tool results with call-order sorting, task tokens, response-format blocks, parse_deepseek_v4 for completions, and bridge_to_next_turn with thinking-retention rules. Prompt text is tokenized in one BPE pass so token IDs match the reference encoder across fragment boundaries.

Registration wires the checkpoint to deepseek-v4 via MODEL_RENDERER_MAP, lazy exports, and the typed config union; README and docs/renderer-config.md document the new family.

Tests add tests/reference_rendering.render_reference so shared parity uses apply_chat_template for Jinja models and an independent V4 mirror for barrage/bridge/config-matrix coverage, plus tests/test_deepseek_v4.py for encoder edge cases.

^{Reviewed by Cursor Bugbot for commit 98cb612. Bugbot is set up for automated code reviews on this repo. Configure here.}

[macroscopeapp]

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This change adds a substantial production renderer and parser for DeepSeek V4, including new thinking, tool-calling, token-attribution, and turn-bridging behavior. Its scope and runtime protocol impact exceed a small bounded addition and merit human review.

Notes:

• Macroscope's correctness review did not run, so approvability was decided on eligibility alone.

^{You can add or adjust custom eligibility rules. Learn more.}

[garrett361]

Thank you @hallerite ! This all looks reasonable to me, but probably someone who is more familiar w/ renderers should give the approval

[garrett361]

Little confusing about what the intended default is: "high" https://api-docs.deepseek.com/guides/thinking_mode/ seems to say "high" is the intended default, but I see it's None = low in the hf impl https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/main/encoding/encoding_dsv4.py#L222

[hallerite]

imo the default should be low, as we are matching against the deepseek encoder, while high being the default on the API is a service-level policy

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a7217e5. Configure here.}