fix: Preserve Kimi tool IDs and call boundaries

AGENTS.md

@@ -6,3 +6,5 @@
 - A request to port, convert, or add a model family to `UnifiedParser` means a full conversion. `UnifiedParser` and its shared scanner must become the only parser state machine for that family. Remove the old bespoke v2 parser state machine; retain a legacy `ToolParser` entry point only when external callers still require it, and make it a thin projection of the same `UnifiedParser` events. Do not report a factory or registry entry as a conversion while a second family-specific buffer, scanner, drain loop, or grammar owner remains.
 - A UnifiedParser conversion is incomplete until its current crate-version capture has been collected: generate the authored Unified inputs and golden events, run the live Unified capture, publish plain-version YAML and required manifest changes, extract the snapshot, and render `conformance/CONFORMANCE_v2.html`. Do not generate `CONFORMANCE_unified.html`.
 - Before reporting a UnifiedParser conversion complete, run `conformance/utils/check.sh status --model <family> --tab unified`. It must report zero red and zero empty current Dynamo cells. Also run the native parser tests, Unified golden parity, the live-vs-packaged-capture guard, and the v2 table-model tests. A generated loose feed, a parser unit test, or a red diagnostic capture is not completion.
+
+- Before adding conformance cases, inspect every registered family and author family-native inputs and expected outputs wherever the behavior applies. Explain grammar or mode exclusions; a bug found in one family does not justify family-only coverage. Use numeric Unified groups for shared behavior and named family sections for grammar-specific cases, preserve published IDs, and inventory every new conformance ID with its meaning and applicable families in the PR description.

conformance/Cargo.toml

@@ -11,7 +11,7 @@ publish = false
 [dev-dependencies]
 anyhow = { workspace = true }
 dynamo-parsers = { workspace = true }
-dynamo-parsers-v2 = { path = "../parsers/v2", version = "0.7.14" }
+dynamo-parsers-v2 = { path = "../parsers/v2", version = "0.7.15" }
 dynamo-protocols = { workspace = true }
 futures = { workspace = true }
 serde = { workspace = true, features = ["derive"] }

conformance/README.md

@@ -31,6 +31,10 @@ When adding or converting a parser, or changing its captured behavior, manually
 
 The release workflow honors this [manual version override](../RELEASING.md#manual-version-peg-fixture-synced-releases): it checks compatibility and publishes the explicitly pinned version without another automatic bump. For #234, GLM Unified and its pending release are both `0.7.0`; GLM Unified has no `0.6.x` captures. Unchanged families continue to inherit their existing captures.
 
+## Case applicability
+
+Before adding a conformance case, inspect every registered family. Shared behaviors need family-native inputs and authored expected outputs for every applicable family. Explain genuine exclusions using the grammar or request-mode limitation; discovering a bug in one family does not justify limiting its coverage to that family. Use numeric Unified groups for shared behaviors and named family sections for cases tied to one family's grammar. Preserve previously published IDs. List every newly added conformance ID, its meaning, applicable families, and intentional exclusions in the PR description.
+
 ## Ownership
 
 Parser v1/v2 terminology, migration steps, and fixture ownership are documented in [`../docs/PARSERS-V2-MIGRATION-PLAN.md`](../docs/PARSERS-V2-MIGRATION-PLAN.md). New streaming parser authors should also read [`../parsers/v2/README.md`](../parsers/v2/README.md); it explains the vLLM-shaped Rust parser contract, the v2 fixture schema, and the exact `conformance/toolcalling/*` files to add. This README covers conformance layout, render outputs, and test commands.
@@ -156,6 +160,8 @@ The commands below cover both the older archive-backed tests and Unified. Apply
 
 ### 2. Fix a Dynamo parser and refresh its expected outputs
 
+For each conformance fix, add a regression test column that shows red for the previous (before) version and green for the fixed (after) version in the rendered report. Capture both versions using the same input, tools, initialization, and chunk schedule, and compare both against the same authored GOLDEN. Preserve the delivery pattern that triggers the bug, including single-chunk input when chunking affects the failure.
+
 1. Fix the code under `parsers/v1/` or `parsers/v2/`.
 2. `cargo test --workspace` — if the fix changes output, the parity tests FAIL. That is the regression gate working: decide whether the diff is a bug in your fix or an intended behavior change.
 3. For an intended Unified change, use the plain-version YAML contract above. Keep the first capture's source origin in metadata; do not create another identity when its SHA changes. For the older tests, retain their existing capture format.

conformance/fixtures-manifest.json

@@ -1,9 +1,9 @@
 {
-  "snapshot": "20261005_101245",
-  "created_pt": "2026-10-05 10:12:45 America/Los_Angeles",
+  "snapshot": "20261006_075805",
+  "created_pt": "2026-10-06 07:58:05 America/Los_Angeles",
   "crates": {
     "dynamo-parsers": "9.2.8",
-    "dynamo-parsers-v2": "0.7.14"
+    "dynamo-parsers-v2": "0.7.15"
   },
   "peers": {
     "vllm": "0.26.0",
@@ -298,8 +298,8 @@
     {
       "path": "unified-history",
       "format": "unified-history",
-      "sha256": "fb273c75596d8f67c7033bc29906a0159879679359403e00ca58329b49677b94",
-      "size": 3877296
+      "sha256": "e89ce39237eb1b51a232f2056d38dd5e1a236dd69999d2bf81e0c43e4c5efedd",
+      "size": 3904011
     }
   ],
   "inactive_shards": []

conformance/fixtures-unified-v2/families/deepseek_v4/dynamo_v2-0.7.13.yaml

@@ -0,0 +1,58 @@
+provenance:
+  status: captured
+  origin:
+    crate_version: 0.7.13
+    source_sha256: f1536c9a9fd7ef3999bfb54e7f3e4d316d5df5c558d43ef9959f1668060f2cf8
+    git_commit: 47ef952237deefe0643f94025272a558f249687d
+document:
+  mode: unified
+changes:
+  malformed_json_then_two_valid_calls:
+    case_key: UNIFIED.5-4
+    stimulus:
+      ref: current
+      semantic_sha256: 47d44fbc177d4fa2972e154079aabd372c389373bbcc57b94f1c888a86ef9ec0
+    observation:
+      value:
+        assembled:
+        - kind: tool_call
+          name: bad
+          arguments:
+            value: '{"x":"unfinished'
+        - kind: tool_call
+          name: echo
+          arguments:
+            value: é
+        - kind: tool_call
+          name: echo
+          arguments:
+            value: Café
+        chunks:
+        - expected:
+          - kind: tool_call
+            name: bad
+            arguments: ''
+            complete: false
+          - kind: tool_call
+            name: null
+            arguments: '{"value":"{\"x\":\"unfinished"}'
+            complete: true
+          - kind: tool_call
+            name: echo
+            arguments: ''
+            complete: false
+          - kind: tool_call
+            name: null
+            arguments: '{"value":"é"}'
+            complete: true
+          - kind: tool_call
+            name: echo
+            arguments: ''
+            complete: false
+          - kind: tool_call
+            name: null
+            arguments: '{"value":"Café"}'
+            complete: true
+        - expected: []
+metadata_changes: {}
+document_overrides: {}

conformance/fixtures-unified-v2/families/deepseek_v4/inputs_and_golden.yaml

@@ -6830,3 +6830,52 @@ cases:
         name: get_weather
         arguments:
           city: 'null'
+  malformed_json_then_two_valid_calls:
+    lifecycle: active
+    scenario: malformed_json_then_two_valid_calls
+    description: 'An unfinished JSON string with explicit call closers precedes echo(value=é) and echo(value=Café). Deliver the entire native input in one push, then finish: marker-aligned delivery masks the Kimi K3 0.7.13 lost-call defect. Preserve family malformed-value fallbacks and both later calls.'
+    policy:
+    - I2
+    - I7
+    - P2
+    display_id: UNIFIED.5-4
+    historical_ids: []
+    request:
+      input: <｜DSML｜tool_calls><｜DSML｜invoke name="bad"><｜DSML｜parameter name="value" string="false">{"x":"unfinished</｜DSML｜parameter></｜DSML｜invoke></｜DSML｜tool_calls><｜DSML｜tool_calls><｜DSML｜invoke name="echo"><｜DSML｜parameter name="value" string="true">é</｜DSML｜parameter></｜DSML｜invoke></｜DSML｜tool_calls><｜DSML｜tool_calls><｜DSML｜invoke name="echo"><｜DSML｜parameter name="value" string="true">Café</｜DSML｜parameter></｜DSML｜invoke></｜DSML｜tool_calls>
+      init:
+        starting_state: None
+        tool_output_mode: Native
+        named_tool: null
+      finish_reason: stop
+      tools:
+      - name: bad
+        parameters:
+          type: object
+          properties:
+            value:
+              type: object
+      - name: echo
+        parameters:
+          type: object
+          properties:
+            value:
+              type: string
+          required:
+          - value
+      chunks:
+      - delta_text: <｜DSML｜tool_calls><｜DSML｜invoke name="bad"><｜DSML｜parameter name="value" string="false">{"x":"unfinished</｜DSML｜parameter></｜DSML｜invoke></｜DSML｜tool_calls><｜DSML｜tool_calls><｜DSML｜invoke name="echo"><｜DSML｜parameter name="value" string="true">é</｜DSML｜parameter></｜DSML｜invoke></｜DSML｜tool_calls><｜DSML｜tool_calls><｜DSML｜invoke name="echo"><｜DSML｜parameter name="value" string="true">Café</｜DSML｜parameter></｜DSML｜invoke></｜DSML｜tool_calls>
+      - delta_text: ‹finish›
+    golden:
+      assembled:
+      - kind: tool_call
+        name: bad
+        arguments:
+          value: '{"x":"unfinished'
+      - kind: tool_call
+        name: echo
+        arguments:
+          value: é
+      - kind: tool_call
+        name: echo
+        arguments:
+          value: Café

conformance/fixtures-unified-v2/families/deepseek_v41/dynamo_v2-0.7.13.yaml

@@ -0,0 +1,20 @@
+provenance:
+  status: captured
+  origin:
+    crate_version: 0.7.13
+    source_sha256: f1536c9a9fd7ef3999bfb54e7f3e4d316d5df5c558d43ef9959f1668060f2cf8
+    git_commit: 47ef952237deefe0643f94025272a558f249687d
+document:
+  mode: unified
+changes:
+  malformed_json_then_two_valid_calls:
+    case_key: UNIFIED.5-4
+    stimulus:
+      ref: current
+      semantic_sha256: a7f43a11f9fdd37253809b088ccc30a3c4eece02700148d2ec3e70402a1bf7e9
+    observation:
+      error:
+        code: legacy_unclassified
+        detail: 'native push: EOF while parsing a string at line 1 column 16'
+metadata_changes: {}
+document_overrides: {}

conformance/fixtures-unified-v2/families/deepseek_v41/inputs_and_golden.yaml

@@ -6765,3 +6765,48 @@ cases:
         name: get_weather
         arguments:
           city: 'null'
+  malformed_json_then_two_valid_calls:
+    lifecycle: active
+    scenario: malformed_json_then_two_valid_calls
+    description: 'An unfinished JSON string with explicit call closers precedes echo(value=é) and echo(value=Café). Deliver the entire native input in one push, then finish: marker-aligned delivery masks the Kimi K3 0.7.13 lost-call defect. Preserve family malformed-value fallbacks and both later calls.'
+    policy:
+    - I2
+    - I7
+    - P2
+    display_id: UNIFIED.5-4
+    historical_ids: []
+    request:
+      input: <｜DSML｜ calls><｜DSML｜ invoke name="bad"><｜DSML｜ parameter name="value" string="false">{"x":"unfinished</｜DSML｜ parameter></｜DSML｜ invoke></｜DSML｜ calls><｜DSML｜ calls><｜DSML｜ invoke name="echo"><｜DSML｜ parameter name="value" string="true">é</｜DSML｜ parameter></｜DSML｜ invoke></｜DSML｜ calls><｜DSML｜ calls><｜DSML｜ invoke name="echo"><｜DSML｜ parameter name="value" string="true">Café</｜DSML｜ parameter></｜DSML｜ invoke></｜DSML｜ calls>
+      init:
+        starting_state: None
+        tool_output_mode: Native
+        named_tool: null
+      finish_reason: stop
+      tools:
+      - name: bad
+        parameters:
+          type: object
+          properties:
+            value:
+              type: object
+      - name: echo
+        parameters:
+          type: object
+          properties:
+            value:
+              type: string
+          required:
+          - value
+      chunks:
+      - delta_text: <｜DSML｜ calls><｜DSML｜ invoke name="bad"><｜DSML｜ parameter name="value" string="false">{"x":"unfinished</｜DSML｜ parameter></｜DSML｜ invoke></｜DSML｜ calls><｜DSML｜ calls><｜DSML｜ invoke name="echo"><｜DSML｜ parameter name="value" string="true">é</｜DSML｜ parameter></｜DSML｜ invoke></｜DSML｜ calls><｜DSML｜ calls><｜DSML｜ invoke name="echo"><｜DSML｜ parameter name="value" string="true">Café</｜DSML｜ parameter></｜DSML｜ invoke></｜DSML｜ calls>
+      - delta_text: ‹finish›
+    golden:
+      assembled:
+      - kind: tool_call
+        name: echo
+        arguments:
+          value: é
+      - kind: tool_call
+        name: echo
+        arguments:
+          value: Café

conformance/fixtures-unified-v2/families/gemma4/dynamo_v2-0.7.13.yaml

@@ -0,0 +1,22 @@
+provenance:
+  status: captured
+  origin:
+    crate_version: 0.7.13
+    source_sha256: f1536c9a9fd7ef3999bfb54e7f3e4d316d5df5c558d43ef9959f1668060f2cf8
+    git_commit: 47ef952237deefe0643f94025272a558f249687d
+document:
+  mode: unified
+changes:
+  malformed_json_then_two_valid_calls:
+    case_key: UNIFIED.5-4
+    stimulus:
+      ref: current
+      semantic_sha256: 1f44f2373df80ee7f4cd7c29c6a0305efe6064847cb22a0f451574d1ae99196c
+    observation:
+      value:
+        assembled: []
+        chunks:
+        - expected: []
+        - expected: []
+metadata_changes: {}
+document_overrides: {}

conformance/fixtures-unified-v2/families/gemma4/inputs_and_golden.yaml

@@ -7214,3 +7214,48 @@ cases:
         name: get_weather
         arguments:
           city: 'null'
+  malformed_json_then_two_valid_calls:
+    lifecycle: active
+    scenario: malformed_json_then_two_valid_calls
+    description: 'An unfinished JSON string with explicit call closers precedes echo(value=é) and echo(value=Café). Deliver the entire native input in one push, then finish: marker-aligned delivery masks the Kimi K3 0.7.13 lost-call defect. Preserve family malformed-value fallbacks and both later calls.'
+    policy:
+    - I2
+    - I7
+    - P2
+    display_id: UNIFIED.5-4
+    historical_ids: []
+    request:
+      input: <|tool_call>call:bad{value:{x:<|"|>unfinished}}<tool_call|><|tool_call>call:echo{value:<|"|>é<|"|>}<tool_call|><|tool_call>call:echo{value:<|"|>Café<|"|>}<tool_call|>
+      init:
+        starting_state: None
+        tool_output_mode: Native
+        named_tool: null
+      finish_reason: stop
+      tools:
+      - name: bad
+        parameters:
+          type: object
+          properties:
+            value:
+              type: object
+      - name: echo
+        parameters:
+          type: object
+          properties:
+            value:
+              type: string
+          required:
+          - value
+      chunks:
+      - delta_text: <|tool_call>call:bad{value:{x:<|"|>unfinished}}<tool_call|><|tool_call>call:echo{value:<|"|>é<|"|>}<tool_call|><|tool_call>call:echo{value:<|"|>Café<|"|>}<tool_call|>
+      - delta_text: ‹finish›
+    golden:
+      assembled:
+      - kind: tool_call
+        name: echo
+        arguments:
+          value: é
+      - kind: tool_call
+        name: echo
+        arguments:
+          value: Café

conformance/fixtures-unified-v2/families/glm47/dynamo_v2-0.7.13.yaml

@@ -0,0 +1,46 @@
+provenance:
+  status: captured
+  origin:
+    crate_version: 0.7.13
+    source_sha256: f1536c9a9fd7ef3999bfb54e7f3e4d316d5df5c558d43ef9959f1668060f2cf8
+    git_commit: 47ef952237deefe0643f94025272a558f249687d
+document:
+  mode: unified
+changes:
+  malformed_json_then_two_valid_calls:
+    case_key: UNIFIED.5-4
+    stimulus:
+      ref: current
+      semantic_sha256: 84e8a9ea473ff97589606f714a092df4793bce23741607a2d3c4c22f54de67ac
+    observation:
+      value:
+        assembled:
+        - kind: tool_call
+          name: bad
+          arguments:
+            value: '{"x":"unfinished'
+        - kind: tool_call
+          name: echo
+          arguments:
+            value: é
+        - kind: tool_call
+          name: echo
+          arguments:
+            value: Café
+        chunks:
+        - expected:
+          - kind: tool_call
+            name: bad
+            arguments: '{"value":"{\"x\":\"unfinished"}'
+            complete: true
+          - kind: tool_call
+            name: echo
+            arguments: '{"value":"é"}'
+            complete: true
+          - kind: tool_call
+            name: echo
+            arguments: '{"value":"Café"}'
+            complete: true
+        - expected: []
+metadata_changes: {}
+document_overrides: {}