feat(parsers): gemma4 on the v2 UnifiedParser — one ordered stream, with Dynamo request modes

[keivenchang]

Overview:

This PR moves Gemma4 onto the v2 UnifiedParser so reasoning, visible content, and tool calls stay in one ordered stream. Gemma4 only; every other family keeps its current path.

This is standalone on current main. #178, #174, and #194 already provide the shared parser interface, request modes, and guided-streaming contract.

Example (before → after):

Case UNIFIED.12.b — a tool call opens inside a reasoning span:

before: reasoning = "I should check. <|tool_call>"   ← markup leaks into reasoning; the call is lost
after:  reasoning = "I should check. ", tool_call = get_weather, reasoning = " now answer"

Details:

• Gemma4 supplies its markers, emitter, and reasoning specification to the shared ScannerUnified; registration is one family row.
• ReasoningSpec.start_label strips Gemma4's structural thought\n role label instead of exposing it as reasoning text.
• Native recovery keeps lexical call: prose, skips wrapper-shaped data inside argument strings, and resynchronizes after malformed or incomplete wrappers.
• The current capture is dynamo_v2-0.3.3+pr166, separate from the released 0.3.3 identity. Historical columns use immutable sparse overlays, and released base hashes remain unchanged.

Verification:

• cargo fmt -- --check, parser clippy, and conformance clippy pass.
• The v2 parser package passes 450 unit tests plus its integration and doc-test binaries.
• The full conformance package passes. python3 -m pytest conformance/utils/tests passes 191 tests.
• The generated v2 matrix contains all 292 unified cases with no postdates that build placeholders.
• The archive audit verifies 28 manifest hashes and sizes, four released hashes, 12 sparse counts, and the four removed archive names.

Where should the reviewer start?

parsers/v2/src/unified/gemma4.rs, then parsers/v2/src/tool_calling/scan.rs and parsers/v2/src/tool_calling/v1core/gemma4/parser.rs

/coderabbit profile chill

[github-actions]

📊 Conformance matrix rendered — view in CI summary

[keivenchang]

Pushed e3df430a with the remaining Gemma4 v2 review fixes and capture updates.

The parser now preserves lexical call: prose and resynchronizes after malformed or incomplete Gemma wrappers in both tool-only and unified modes at every UTF-8 split. The capture set now keeps released bases immutable, fills historical columns through sparse overlays, and records this PR under 0.3.3+pr166.

Local validation passes: format and clippy, 450 parser unit tests plus integrations/docs, the full conformance package, 191 Python utility tests, and a 292-case matrix with no historical placeholders. CI for e3df430a is still in progress. I published the three prepared replies and left all eight reviewer threads open for Ryan.

[indrajit96]

@keivenchang thanks for adding this
Left a few comments on streaming changes with a possible perf issue.
SOme gemma specific behavior is still present in unified/mod.rs
The existing tests all pass.

[indrajit96]

LGTM!
Conditional approval with one corner case comment here #166 (comment)

[keivenchang]

Indrajit’s conditional approval is addressed in 2a6d200 — the parser now preserves a balanced recovery candidate across trailing whitespace at EOF, with native and unified coverage at chunk sizes 1, 4, and 16.