fix: Preserve Kimi tool IDs and call boundaries

[keivenchang]

Overview:

Preserve native Kimi tool IDs and call boundaries, including K3 recovery when an unfinished raw JSON string hides valid calls that follow it. Recovery waits for EOF, drops the malformed raw-JSON call after its explicit JSON/call closers, and emits the later calls once in order. Complete call-shaped passages inside string arguments stay literal.

This PR focuses on Kimi parser fixes. The conformance report includes known and newly discovered problems in other families, including Gemma4 dropping the following calls and DeepSeek V4.1 returning a JSON parse error. Those failures remain visible in the report and require separate parser fixes.

Details:

before:

after:

• Native and supported Guided required/named routes share the K3 boundary tracker. Checkpoints cover every valid UTF-8 split, scalar delivery, intermediate progress, literal protection, native IDs, EOF timing, and reset/reuse.
• Add UNIFIED.5-4 — malformed_json_then_two_valid_calls to all eight Unified families using each family's native syntax. The request offers bad with an object-valued argument and echo with required string value, starts in the default state, and uses Native output.
• The input contains an unfinished JSON/native string, explicit call closers, then echo(value="é") and echo(value="Café"). It is delivered in one push followed by finish. Published K3 0.7.13 passes with marker-aligned delivery, so the previous schedule missed the lost-call defect.
• Optional authored input_chunks now survives emission, current capture, historical replay, provenance validation, and live-versus-committed checks through one shared helper. Existing cases retain marker-aligned delivery. Public parser APIs are unchanged by this conformance addition.
• Capture errors remain errors and render red. Dynamo report assembly uses the recorded malformed-JSON fallback only after the full call sequence aligns. Native {} recovery and split-era raw-string output both remain accurate; peer output remains unchanged. The three previously removed columns remain absent.

Family	Published 0.7.13	Candidate 0.7.15
Kimi K3	Red: only Café survives	Green: é, then Café
Kimi K2	Green; malformed call retains {}	Same
Qwen3, GLM47, DeepSeek V4, Muse	Green; malformed object value retains literal-string fallback	Same
Gemma4	Red: no calls	Same
DeepSeek V4.1	Red: JSON parse error	Same

Gemma4 and DeepSeek V4.1 still have two-call goldens. Their measured failures are explicitly recorded; fixing those parsers is outside this change. There are no family exclusions for 5-4. The complete Unified corpus contains 113 scenarios and 770 applicable family/case pairs.

Capture identity and report:

• Merged main at f279dcc139f9f00e056d78defde4a4d4e962d679, whose parser version is published 0.7.14. Candidate version 0.7.15 remains unpublished. The earlier unpublished 0.7.14 K3 checkpoint was retired and recaptured against 0.7.15 to avoid claiming candidate output came from the released package.
• Historical observations ran against the actual crates.io dynamo-parsers-v2 = 0.7.13 Unified parser. All 53 published Rust source files match release tag commit 47ef952237deefe0643f94025272a558f249687d. Crate SHA-256: c278da81d4ffc1e629d2c6faa57a966aa66d3767eb04c41839bf1cb4d29b7f03.
• Candidate capture source fingerprint: 8083be6e32dae8e8e337dca363273bf1ad96f5b705f3685a331e468df9292d2d. Its origin records the first capture anchor; this is an unpublished candidate, not a released package.
• Added eight 0.7.13 observations and one changed K3 0.7.15 checkpoint. Seven unchanged family results carry forward. All 67 prior checkpoint documents and prior input/result/origin metadata are unchanged. Historical backcapture for this addition is scoped to 0.7.13; older columns retain their previous unavailable coverage. An additional actual published 0.7.14 probe reproduces the same eight-family results, including K3 losing the first echo in one push and passing with marker-aligned delivery. Its registry checksum is b5b275a1ecd1b50ef5107ec10b10be1ed62b1c4d5f730956b6576c2876116b0a; all Rust source files match release commit f279dcc139f9f00e056d78defde4a4d4e962d679.
• bash conformance/utils/regenerate_unified.sh ran generation, capture, packaging, extraction, and report checks. bash conformance/utils/render_table_v2.sh --output conformance/CONFORMANCE_v2.html regenerated the final HTML/JSON after the display correction. Real Chromium verified the 16 family/version cell colors. Generated reports are not committed.

Verification:

• Final merged candidate: 827 Python conformance tests, 718 parser-library tests, the Unified generation/capture/package/extract/render gate, formatting, and workspace Clippy passed. The incoming MiniMax, legacy conformance, and renderer tests passed. Independent merge audit verified all 83 other prior Unified files remain byte-identical and confirmed the new capture/report identities.

• 718 parser-library tests; 11 schema round-trip, 15 Unified parity, 16 live/capture/render, and 17 historical-capture tests passed.

• 385 model/taxonomy/overlay tests, 102 focused harness/schema/assembly/history tests, 69 provenance/history tests, 108 assembly/model tests, and 14 browser comparison tests passed. These suite counts overlap.

• Formatting and workspace Clippy with all targets/features and warnings denied passed.

• Exact candidate tree bbb3c39d01ce72b5f8a6fbf3811b2fd048b978d7 passed the parser-gate validator. Evidence covers lifecycle, mode entry, split/progress, EOF, payload ownership, schemas, wrappers, producer/consumer parity, goldens, capture identity, corpus applicability, append-only history, and final regeneration. Negative controls include published K3 single-push loss, 32 mutated family stimuli/goldens/schedules, invalid replay/error records, and the prior report assembly/color failures.

• Independent audit verified the capture identities, unchanged history, eight-family matrix, and browser behavior, including unchanged non-oracle error coloring. The final audit scanned 5,290 resolved Dynamo observations; only the intended new K2 recovery observation changes rendering.

Related: DIS-2500

[github-actions]

📊 Conformance matrix rendered — view in CI summary

[keivenchang]

Code reviewed by GPT-6-Astra medium. Reviewed d71012fb after the main merge; no remaining substantive findings in the audited scope. The Kimi K3 literal-call regression is fixed, and the fresh independent harness passed 14,688 Native/Guided and bare/wrapped split checks. Final local validation passed 691 parser tests, 10 schema tests, 14 parity tests, 15 render tests, 364 conformance tests, 94 provenance/history/stimulus tests, 149 protocol tests, and both merged-base MiniMax regressions; formatting and Clippy passed. The regenerated report has all three new IDs, 772 applicable family/case pairs, and zero red or empty current Unified cells across eight families. All 3,087 original historical capture records are preserved.

For follow-up AI reviews: unchanged 0.7.14 outputs intentionally inherit earlier semantic checkpoints, so absent standalone .14 YAML files are expected and covered by the store-policy tests. Recorded historical producer/source/dependency limitations can remain deferred; newer outputs must not replace those observations. The companion Dynamo response-conversion assertions are retained, and their release-dependent integration suite belongs to the companion PR. Do not repeat these documented scope decisions as missing parser fixes without new evidence of incorrect behavior.

Current-head CI passed on d71012fb, including Rust tests, doctests, Clippy, crate publishability, and conformance. No remaining code or CI blocker was found in this review; required human approval still applies. The PR is mergeable and marked Ready for review; its title and description are updated.

[dynamo-review-agent]

Previously reported defects still present:

• Original discussion: A string argument that has closed its argument marker but is missing the owning call close still leaves string_argument set, so the next real call header is classified as literal data and swallowed into the previous argument. Clear the string-literal state once the argument close is known to be structural so subsequent call headers can start recoverable calls.
• Original discussion: The Kimi K3 boundary tracker still turns a completed string argument followed by ordinary response/reasoning prose back into an open string argument, so a call followed by prose is not emitted until EOF instead of when the next structural boundary arrives. Keep real call closes in response/reasoning channels closed when prose follows them.
• Original discussion: The scanner still records call markers found inside object/JSON argument strings as complete call openings, so a string value containing <|close|>call<|sep|><|open|>call<|sep|> can make the parser resync at the embedded header and drop the real outer call or emit spurious calls. Ignore Kimi channel markers while scanning JSON/object string contents.

[dynamo-review-agent]

Previously reported defects still present:

• Original discussion: Still present: adjacent_calls_keep_each_boundary_at_every_split runs count=1, repeated=true, but with only one call the repeated flag only changes the payload value and does not exercise duplicate-call boundary handling. The single-call boundary remains covered by count=1, repeated=false, and identical repeated calls remain covered by retained count > 1 repeated cases.
• Original discussion: Still present: typed_string_header_candidates_do_not_hide_later_calls includes the literal.repeat(3) row, which exercises the same observable contract as the retained two-repeat row: multiple complete call-shaped passages inside a typed string remain data while later real calls survive. The third identical repetition adds no distinct parser state or supported boundary.
• Original discussion: Verified still present: after a string argument closes without its owning call close, argument_close_pending is true but string_argument remains true, so the next real call header is still classified as literal data via pending_call_is_literal = ... || self.string_argument and can be swallowed into the previous argument instead of starting the following call.
• Original discussion: Verified still present: when a completed string argument and call close are followed by ordinary response/reasoning prose, the non-whitespace scan path restores string_argument = true and clears outer_call_closed, so the completed call remains pending until EOF instead of being emitted at the next structural boundary.

[dynamo-review-agent]

Previously reported defects still present:

• Original discussion: Still present: after a string argument closes without its owning call close, argument_close_pending can be true while string_argument remains true, so the next real call header is still classified as literal data via pending_call_is_literal = ... || self.string_argument and can be swallowed into the previous argument instead of starting the following call.
• Original discussion: Still present: when a completed string argument and call close are followed by ordinary response/reasoning prose, the non-whitespace scan path restores string_argument = true and clears outer_call_closed, so the completed call remains pending until EOF instead of being emitted at the next structural boundary.
• Original discussion: Verified still present: after a string argument closes without its owning call close, argument_close_pending is true but string_argument remains true, so the next real call header is classified as literal data and can be swallowed into the previous argument instead of starting the following call. Clear the string-literal state once the argument close is known to be structural so subsequent call headers can start recoverable calls.
• Original discussion: Verified still present: when a completed string argument and call close are followed by ordinary response/reasoning prose, the non-whitespace scan path restores string_argument = true and clears outer_call_closed, so the completed call remains pending until EOF instead of being emitted at the next structural boundary. Keep real call closes in response/reasoning channels closed when prose follows them.

[keivenchang]

Replying to AI reviews 5420727540, 5421163515, and 5421264621: rechecked their five repeated findings on 0a4c960f. Missing-close recovery and prose between calls emit the exact calls/arguments/IDs at the enclosing fence before EOF; delaying beyond the next header remains intentional to protect literal passages. JSON-marker ownership preserves the outer object and following call without ghost calls. Both requested single-call and literal-repetition test reductions are already present. The 72 K3 tests and an independent focused audit pass. The uploaded commit adds the requested conformance documentation; no further parser change was warranted by these repeated claims.