diff --git a/AGENTS.md b/AGENTS.md index 143513bbe..f715fe292 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -6,3 +6,5 @@ - A request to port, convert, or add a model family to `UnifiedParser` means a full conversion. `UnifiedParser` and its shared scanner must become the only parser state machine for that family. Remove the old bespoke v2 parser state machine; retain a legacy `ToolParser` entry point only when external callers still require it, and make it a thin projection of the same `UnifiedParser` events. Do not report a factory or registry entry as a conversion while a second family-specific buffer, scanner, drain loop, or grammar owner remains. - A UnifiedParser conversion is incomplete until its current crate-version capture has been collected: generate the authored Unified inputs and golden events, run the live Unified capture, publish plain-version YAML and required manifest changes, extract the snapshot, and render `conformance/CONFORMANCE_v2.html`. Do not generate `CONFORMANCE_unified.html`. - Before reporting a UnifiedParser conversion complete, run `conformance/utils/check.sh status --model --tab unified`. It must report zero red and zero empty current Dynamo cells. Also run the native parser tests, Unified golden parity, the live-vs-packaged-capture guard, and the v2 table-model tests. A generated loose feed, a parser unit test, or a red diagnostic capture is not completion. + +- Before adding conformance cases, inspect every registered family and author family-native inputs and expected outputs wherever the behavior applies. Explain grammar or mode exclusions; a bug found in one family does not justify family-only coverage. Use numeric Unified groups for shared behavior and named family sections for grammar-specific cases, preserve published IDs, and inventory every new conformance ID with its meaning and applicable families in the PR description. diff --git a/Cargo.lock b/Cargo.lock index af13634cc..9ad41de86 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1178,7 +1178,7 @@ dependencies = [ [[package]] name = "dynamo-parsers-v2" -version = "0.7.14" +version = "0.7.15" dependencies = [ "anyhow", "openai-harmony", diff --git a/conformance/Cargo.toml b/conformance/Cargo.toml index 32e715aaa..2a45b61e0 100644 --- a/conformance/Cargo.toml +++ b/conformance/Cargo.toml @@ -11,7 +11,7 @@ publish = false [dev-dependencies] anyhow = { workspace = true } dynamo-parsers = { workspace = true } -dynamo-parsers-v2 = { path = "../parsers/v2", version = "0.7.14" } +dynamo-parsers-v2 = { path = "../parsers/v2", version = "0.7.15" } dynamo-protocols = { workspace = true } futures = { workspace = true } serde = { workspace = true, features = ["derive"] } diff --git a/conformance/README.md b/conformance/README.md index 4be8a4315..33be658da 100644 --- a/conformance/README.md +++ b/conformance/README.md @@ -31,6 +31,10 @@ When adding or converting a parser, or changing its captured behavior, manually The release workflow honors this [manual version override](../RELEASING.md#manual-version-peg-fixture-synced-releases): it checks compatibility and publishes the explicitly pinned version without another automatic bump. For #234, GLM Unified and its pending release are both `0.7.0`; GLM Unified has no `0.6.x` captures. Unchanged families continue to inherit their existing captures. +## Case applicability + +Before adding a conformance case, inspect every registered family. Shared behaviors need family-native inputs and authored expected outputs for every applicable family. Explain genuine exclusions using the grammar or request-mode limitation; discovering a bug in one family does not justify limiting its coverage to that family. Use numeric Unified groups for shared behaviors and named family sections for cases tied to one family's grammar. Preserve previously published IDs. List every newly added conformance ID, its meaning, applicable families, and intentional exclusions in the PR description. + ## Ownership Parser v1/v2 terminology, migration steps, and fixture ownership are documented in [`../docs/PARSERS-V2-MIGRATION-PLAN.md`](../docs/PARSERS-V2-MIGRATION-PLAN.md). New streaming parser authors should also read [`../parsers/v2/README.md`](../parsers/v2/README.md); it explains the vLLM-shaped Rust parser contract, the v2 fixture schema, and the exact `conformance/toolcalling/*` files to add. This README covers conformance layout, render outputs, and test commands. @@ -156,6 +160,8 @@ The commands below cover both the older archive-backed tests and Unified. Apply ### 2. Fix a Dynamo parser and refresh its expected outputs +For each conformance fix, add a regression test column that shows red for the previous (before) version and green for the fixed (after) version in the rendered report. Capture both versions using the same input, tools, initialization, and chunk schedule, and compare both against the same authored GOLDEN. Preserve the delivery pattern that triggers the bug, including single-chunk input when chunking affects the failure. + 1. Fix the code under `parsers/v1/` or `parsers/v2/`. 2. `cargo test --workspace` — if the fix changes output, the parity tests FAIL. That is the regression gate working: decide whether the diff is a bug in your fix or an intended behavior change. 3. For an intended Unified change, use the plain-version YAML contract above. Keep the first capture's source origin in metadata; do not create another identity when its SHA changes. For the older tests, retain their existing capture format. diff --git a/conformance/fixtures-manifest.json b/conformance/fixtures-manifest.json index 121b18e2e..cfd6dbde4 100644 --- a/conformance/fixtures-manifest.json +++ b/conformance/fixtures-manifest.json @@ -1,9 +1,9 @@ { - "snapshot": "20261005_101245", - "created_pt": "2026-10-05 10:12:45 America/Los_Angeles", + "snapshot": "20261006_075805", + "created_pt": "2026-10-06 07:58:05 America/Los_Angeles", "crates": { "dynamo-parsers": "9.2.8", - "dynamo-parsers-v2": "0.7.14" + "dynamo-parsers-v2": "0.7.15" }, "peers": { "vllm": "0.26.0", @@ -298,8 +298,8 @@ { "path": "unified-history", "format": "unified-history", - "sha256": "fb273c75596d8f67c7033bc29906a0159879679359403e00ca58329b49677b94", - "size": 3877296 + "sha256": "e89ce39237eb1b51a232f2056d38dd5e1a236dd69999d2bf81e0c43e4c5efedd", + "size": 3904011 } ], "inactive_shards": [] diff --git a/conformance/fixtures-unified-v2/families/deepseek_v4/dynamo_v2-0.7.13.yaml b/conformance/fixtures-unified-v2/families/deepseek_v4/dynamo_v2-0.7.13.yaml new file mode 100644 index 000000000..585ccb4a3 --- /dev/null +++ b/conformance/fixtures-unified-v2/families/deepseek_v4/dynamo_v2-0.7.13.yaml @@ -0,0 +1,58 @@ +provenance: + status: captured + origin: + crate_version: 0.7.13 + source_sha256: f1536c9a9fd7ef3999bfb54e7f3e4d316d5df5c558d43ef9959f1668060f2cf8 + git_commit: 47ef952237deefe0643f94025272a558f249687d +document: + mode: unified +changes: + malformed_json_then_two_valid_calls: + case_key: UNIFIED.5-4 + stimulus: + ref: current + semantic_sha256: 47d44fbc177d4fa2972e154079aabd372c389373bbcc57b94f1c888a86ef9ec0 + observation: + value: + assembled: + - kind: tool_call + name: bad + arguments: + value: '{"x":"unfinished' + - kind: tool_call + name: echo + arguments: + value: é + - kind: tool_call + name: echo + arguments: + value: Café + chunks: + - expected: + - kind: tool_call + name: bad + arguments: '' + complete: false + - kind: tool_call + name: null + arguments: '{"value":"{\"x\":\"unfinished"}' + complete: true + - kind: tool_call + name: echo + arguments: '' + complete: false + - kind: tool_call + name: null + arguments: '{"value":"é"}' + complete: true + - kind: tool_call + name: echo + arguments: '' + complete: false + - kind: tool_call + name: null + arguments: '{"value":"Café"}' + complete: true + - expected: [] +metadata_changes: {} +document_overrides: {} diff --git a/conformance/fixtures-unified-v2/families/deepseek_v4/inputs_and_golden.yaml b/conformance/fixtures-unified-v2/families/deepseek_v4/inputs_and_golden.yaml index ca21c6a6c..51419e540 100644 --- a/conformance/fixtures-unified-v2/families/deepseek_v4/inputs_and_golden.yaml +++ b/conformance/fixtures-unified-v2/families/deepseek_v4/inputs_and_golden.yaml @@ -6830,3 +6830,52 @@ cases: name: get_weather arguments: city: 'null' + malformed_json_then_two_valid_calls: + lifecycle: active + scenario: malformed_json_then_two_valid_calls + description: 'An unfinished JSON string with explicit call closers precedes echo(value=é) and echo(value=Café). Deliver the entire native input in one push, then finish: marker-aligned delivery masks the Kimi K3 0.7.13 lost-call defect. Preserve family malformed-value fallbacks and both later calls.' + policy: + - I2 + - I7 + - P2 + display_id: UNIFIED.5-4 + historical_ids: [] + request: + input: <｜DSML｜tool_calls><｜DSML｜invoke name="bad"><｜DSML｜parameter name="value" string="false">{"x":"unfinished<｜DSML｜tool_calls><｜DSML｜invoke name="echo"><｜DSML｜parameter name="value" string="true">é<｜DSML｜tool_calls><｜DSML｜invoke name="echo"><｜DSML｜parameter name="value" string="true">Café + init: + starting_state: None + tool_output_mode: Native + named_tool: null + finish_reason: stop + tools: + - name: bad + parameters: + type: object + properties: + value: + type: object + - name: echo + parameters: + type: object + properties: + value: + type: string + required: + - value + chunks: + - delta_text: <｜DSML｜tool_calls><｜DSML｜invoke name="bad"><｜DSML｜parameter name="value" string="false">{"x":"unfinished<｜DSML｜tool_calls><｜DSML｜invoke name="echo"><｜DSML｜parameter name="value" string="true">é<｜DSML｜tool_calls><｜DSML｜invoke name="echo"><｜DSML｜parameter name="value" string="true">Café + - delta_text: ‹finish› + golden: + assembled: + - kind: tool_call + name: bad + arguments: + value: '{"x":"unfinished' + - kind: tool_call + name: echo + arguments: + value: é + - kind: tool_call + name: echo + arguments: + value: Café diff --git a/conformance/fixtures-unified-v2/families/deepseek_v41/dynamo_v2-0.7.13.yaml b/conformance/fixtures-unified-v2/families/deepseek_v41/dynamo_v2-0.7.13.yaml new file mode 100644 index 000000000..4f3cd9e77 --- /dev/null +++ b/conformance/fixtures-unified-v2/families/deepseek_v41/dynamo_v2-0.7.13.yaml @@ -0,0 +1,20 @@ +provenance: + status: captured + origin: + crate_version: 0.7.13 + source_sha256: f1536c9a9fd7ef3999bfb54e7f3e4d316d5df5c558d43ef9959f1668060f2cf8 + git_commit: 47ef952237deefe0643f94025272a558f249687d +document: + mode: unified +changes: + malformed_json_then_two_valid_calls: + case_key: UNIFIED.5-4 + stimulus: + ref: current + semantic_sha256: a7f43a11f9fdd37253809b088ccc30a3c4eece02700148d2ec3e70402a1bf7e9 + observation: + error: + code: legacy_unclassified + detail: 'native push: EOF while parsing a string at line 1 column 16' +metadata_changes: {} +document_overrides: {} diff --git a/conformance/fixtures-unified-v2/families/deepseek_v41/inputs_and_golden.yaml b/conformance/fixtures-unified-v2/families/deepseek_v41/inputs_and_golden.yaml index 7fdd6e81e..b27174811 100644 --- a/conformance/fixtures-unified-v2/families/deepseek_v41/inputs_and_golden.yaml +++ b/conformance/fixtures-unified-v2/families/deepseek_v41/inputs_and_golden.yaml @@ -6765,3 +6765,48 @@ cases: name: get_weather arguments: city: 'null' + malformed_json_then_two_valid_calls: + lifecycle: active + scenario: malformed_json_then_two_valid_calls + description: 'An unfinished JSON string with explicit call closers precedes echo(value=é) and echo(value=Café). Deliver the entire native input in one push, then finish: marker-aligned delivery masks the Kimi K3 0.7.13 lost-call defect. Preserve family malformed-value fallbacks and both later calls.' + policy: + - I2 + - I7 + - P2 + display_id: UNIFIED.5-4 + historical_ids: [] + request: + input: <｜DSML｜ calls><｜DSML｜ invoke name="bad"><｜DSML｜ parameter name="value" string="false">{"x":"unfinished<｜DSML｜ calls><｜DSML｜ invoke name="echo"><｜DSML｜ parameter name="value" string="true">é<｜DSML｜ calls><｜DSML｜ invoke name="echo"><｜DSML｜ parameter name="value" string="true">Café + init: + starting_state: None + tool_output_mode: Native + named_tool: null + finish_reason: stop + tools: + - name: bad + parameters: + type: object + properties: + value: + type: object + - name: echo + parameters: + type: object + properties: + value: + type: string + required: + - value + chunks: + - delta_text: <｜DSML｜ calls><｜DSML｜ invoke name="bad"><｜DSML｜ parameter name="value" string="false">{"x":"unfinished<｜DSML｜ calls><｜DSML｜ invoke name="echo"><｜DSML｜ parameter name="value" string="true">é<｜DSML｜ calls><｜DSML｜ invoke name="echo"><｜DSML｜ parameter name="value" string="true">Café + - delta_text: ‹finish› + golden: + assembled: + - kind: tool_call + name: echo + arguments: + value: é + - kind: tool_call + name: echo + arguments: + value: Café diff --git a/conformance/fixtures-unified-v2/families/gemma4/dynamo_v2-0.7.13.yaml b/conformance/fixtures-unified-v2/families/gemma4/dynamo_v2-0.7.13.yaml new file mode 100644 index 000000000..a12b4773c --- /dev/null +++ b/conformance/fixtures-unified-v2/families/gemma4/dynamo_v2-0.7.13.yaml @@ -0,0 +1,22 @@ +provenance: + status: captured + origin: + crate_version: 0.7.13 + source_sha256: f1536c9a9fd7ef3999bfb54e7f3e4d316d5df5c558d43ef9959f1668060f2cf8 + git_commit: 47ef952237deefe0643f94025272a558f249687d +document: + mode: unified +changes: + malformed_json_then_two_valid_calls: + case_key: UNIFIED.5-4 + stimulus: + ref: current + semantic_sha256: 1f44f2373df80ee7f4cd7c29c6a0305efe6064847cb22a0f451574d1ae99196c + observation: + value: + assembled: [] + chunks: + - expected: [] + - expected: [] +metadata_changes: {} +document_overrides: {} diff --git a/conformance/fixtures-unified-v2/families/gemma4/inputs_and_golden.yaml b/conformance/fixtures-unified-v2/families/gemma4/inputs_and_golden.yaml index 38f43c89d..32bbd8329 100644 --- a/conformance/fixtures-unified-v2/families/gemma4/inputs_and_golden.yaml +++ b/conformance/fixtures-unified-v2/families/gemma4/inputs_and_golden.yaml @@ -7214,3 +7214,48 @@ cases: name: get_weather arguments: city: 'null' + malformed_json_then_two_valid_calls: + lifecycle: active + scenario: malformed_json_then_two_valid_calls + description: 'An unfinished JSON string with explicit call closers precedes echo(value=é) and echo(value=Café). Deliver the entire native input in one push, then finish: marker-aligned delivery masks the Kimi K3 0.7.13 lost-call defect. Preserve family malformed-value fallbacks and both later calls.' + policy: + - I2 + - I7 + - P2 + display_id: UNIFIED.5-4 + historical_ids: [] + request: + input: <|tool_call>call:bad{value:{x:<|"|>unfinished}}<|tool_call>call:echo{value:<|"|>é<|"|>}<|tool_call>call:echo{value:<|"|>Café<|"|>} + init: + starting_state: None + tool_output_mode: Native + named_tool: null + finish_reason: stop + tools: + - name: bad + parameters: + type: object + properties: + value: + type: object + - name: echo + parameters: + type: object + properties: + value: + type: string + required: + - value + chunks: + - delta_text: <|tool_call>call:bad{value:{x:<|"|>unfinished}}<|tool_call>call:echo{value:<|"|>é<|"|>}<|tool_call>call:echo{value:<|"|>Café<|"|>} + - delta_text: ‹finish› + golden: + assembled: + - kind: tool_call + name: echo + arguments: + value: é + - kind: tool_call + name: echo + arguments: + value: Café diff --git a/conformance/fixtures-unified-v2/families/glm47/dynamo_v2-0.7.13.yaml b/conformance/fixtures-unified-v2/families/glm47/dynamo_v2-0.7.13.yaml new file mode 100644 index 000000000..6a514d491 --- /dev/null +++ b/conformance/fixtures-unified-v2/families/glm47/dynamo_v2-0.7.13.yaml @@ -0,0 +1,46 @@ +provenance: + status: captured + origin: + crate_version: 0.7.13 + source_sha256: f1536c9a9fd7ef3999bfb54e7f3e4d316d5df5c558d43ef9959f1668060f2cf8 + git_commit: 47ef952237deefe0643f94025272a558f249687d +document: + mode: unified +changes: + malformed_json_then_two_valid_calls: + case_key: UNIFIED.5-4 + stimulus: + ref: current + semantic_sha256: 84e8a9ea473ff97589606f714a092df4793bce23741607a2d3c4c22f54de67ac + observation: + value: + assembled: + - kind: tool_call + name: bad + arguments: + value: '{"x":"unfinished' + - kind: tool_call + name: echo + arguments: + value: é + - kind: tool_call + name: echo + arguments: + value: Café + chunks: + - expected: + - kind: tool_call + name: bad + arguments: '{"value":"{\"x\":\"unfinished"}' + complete: true + - kind: tool_call + name: echo + arguments: '{"value":"é"}' + complete: true + - kind: tool_call + name: echo + arguments: '{"value":"Café"}' + complete: true + - expected: [] +metadata_changes: {} +document_overrides: {} diff --git a/conformance/fixtures-unified-v2/families/glm47/inputs_and_golden.yaml b/conformance/fixtures-unified-v2/families/glm47/inputs_and_golden.yaml index bda97f0ac..d20f50b56 100644 --- a/conformance/fixtures-unified-v2/families/glm47/inputs_and_golden.yaml +++ b/conformance/fixtures-unified-v2/families/glm47/inputs_and_golden.yaml @@ -7025,3 +7025,52 @@ cases: arguments: payload: x: 1 + malformed_json_then_two_valid_calls: + lifecycle: active + scenario: malformed_json_then_two_valid_calls + description: 'An unfinished JSON string with explicit call closers precedes echo(value=é) and echo(value=Café). Deliver the entire native input in one push, then finish: marker-aligned delivery masks the Kimi K3 0.7.13 lost-call defect. Preserve family malformed-value fallbacks and both later calls.' + policy: + - I2 + - I7 + - P2 + display_id: UNIFIED.5-4 + historical_ids: [] + request: + input: badvalue{"x":"unfinishedechovalueéechovalueCafé + init: + starting_state: None + tool_output_mode: Native + named_tool: null + finish_reason: stop + tools: + - name: bad + parameters: + type: object + properties: + value: + type: object + - name: echo + parameters: + type: object + properties: + value: + type: string + required: + - value + chunks: + - delta_text: badvalue{"x":"unfinishedechovalueéechovalueCafé + - delta_text: ‹finish› + golden: + assembled: + - kind: tool_call + name: bad + arguments: + value: '{"x":"unfinished' + - kind: tool_call + name: echo + arguments: + value: é + - kind: tool_call + name: echo + arguments: + value: Café diff --git a/conformance/fixtures-unified-v2/families/kimi_k2/dynamo_v2-0.7.13.yaml b/conformance/fixtures-unified-v2/families/kimi_k2/dynamo_v2-0.7.13.yaml new file mode 100644 index 000000000..72f2db3d6 --- /dev/null +++ b/conformance/fixtures-unified-v2/families/kimi_k2/dynamo_v2-0.7.13.yaml @@ -0,0 +1,45 @@ +provenance: + status: captured + origin: + crate_version: 0.7.13 + source_sha256: f1536c9a9fd7ef3999bfb54e7f3e4d316d5df5c558d43ef9959f1668060f2cf8 + git_commit: 47ef952237deefe0643f94025272a558f249687d +document: + mode: unified +changes: + malformed_json_then_two_valid_calls: + case_key: UNIFIED.5-4 + stimulus: + ref: current + semantic_sha256: 666722533692b40bd0c3c91ad00937a8189aac7dbe6d1989c32e4490d9c7823c + observation: + value: + assembled: + - kind: tool_call + name: bad + arguments: {} + - kind: tool_call + name: echo + arguments: + value: é + - kind: tool_call + name: echo + arguments: + value: Café + chunks: + - expected: + - kind: tool_call + name: bad + arguments: '{"x":"unfinished' + complete: true + - kind: tool_call + name: echo + arguments: '{"value":"é"}' + complete: true + - kind: tool_call + name: echo + arguments: '{"value":"Café"}' + complete: true + - expected: [] +metadata_changes: {} +document_overrides: {} diff --git a/conformance/fixtures-unified-v2/families/kimi_k2/inputs_and_golden.yaml b/conformance/fixtures-unified-v2/families/kimi_k2/inputs_and_golden.yaml index 099c412ab..357a54d19 100644 --- a/conformance/fixtures-unified-v2/families/kimi_k2/inputs_and_golden.yaml +++ b/conformance/fixtures-unified-v2/families/kimi_k2/inputs_and_golden.yaml @@ -6878,3 +6878,51 @@ cases: name: get_weather arguments: city: 'null' + malformed_json_then_two_valid_calls: + lifecycle: active + scenario: malformed_json_then_two_valid_calls + description: 'An unfinished JSON string with explicit call closers precedes echo(value=é) and echo(value=Café). Deliver the entire native input in one push, then finish: marker-aligned delivery masks the Kimi K3 0.7.13 lost-call defect. Preserve family malformed-value fallbacks and both later calls.' + policy: + - I2 + - I7 + - P2 + display_id: UNIFIED.5-4 + historical_ids: [] + request: + input: '<|tool_calls_section_begin|><|tool_call_begin|>functions.bad:0<|tool_call_argument_begin|>{"x":"unfinished<|tool_call_end|><|tool_calls_section_end|><|tool_calls_section_begin|><|tool_call_begin|>functions.echo:1<|tool_call_argument_begin|>{"value": "é"}<|tool_call_end|><|tool_calls_section_end|><|tool_calls_section_begin|><|tool_call_begin|>functions.echo:2<|tool_call_argument_begin|>{"value": "Café"}<|tool_call_end|><|tool_calls_section_end|>' + init: + starting_state: None + tool_output_mode: Native + named_tool: null + finish_reason: stop + tools: + - name: bad + parameters: + type: object + properties: + value: + type: object + - name: echo + parameters: + type: object + properties: + value: + type: string + required: + - value + chunks: + - delta_text: '<|tool_calls_section_begin|><|tool_call_begin|>functions.bad:0<|tool_call_argument_begin|>{"x":"unfinished<|tool_call_end|><|tool_calls_section_end|><|tool_calls_section_begin|><|tool_call_begin|>functions.echo:1<|tool_call_argument_begin|>{"value": "é"}<|tool_call_end|><|tool_calls_section_end|><|tool_calls_section_begin|><|tool_call_begin|>functions.echo:2<|tool_call_argument_begin|>{"value": "Café"}<|tool_call_end|><|tool_calls_section_end|>' + - delta_text: ‹finish› + golden: + assembled: + - kind: tool_call + name: bad + arguments: {} + - kind: tool_call + name: echo + arguments: + value: é + - kind: tool_call + name: echo + arguments: + value: Café diff --git a/conformance/fixtures-unified-v2/families/kimi_k3/dynamo_v2-0.7.13.yaml b/conformance/fixtures-unified-v2/families/kimi_k3/dynamo_v2-0.7.13.yaml new file mode 100644 index 000000000..646ae4de9 --- /dev/null +++ b/conformance/fixtures-unified-v2/families/kimi_k3/dynamo_v2-0.7.13.yaml @@ -0,0 +1,30 @@ +provenance: + status: captured + origin: + crate_version: 0.7.13 + source_sha256: f1536c9a9fd7ef3999bfb54e7f3e4d316d5df5c558d43ef9959f1668060f2cf8 + git_commit: 47ef952237deefe0643f94025272a558f249687d +document: + mode: unified +changes: + malformed_json_then_two_valid_calls: + case_key: UNIFIED.5-4 + stimulus: + ref: current + semantic_sha256: 0787ab8c2229e3c2d6391cdb11e439f4e66a0f0fb22982c92550526404a2caee + observation: + value: + assembled: + - kind: tool_call + name: echo + arguments: + value: Café + chunks: + - expected: + - kind: tool_call + name: echo + arguments: '{"value":"Café"}' + complete: true + - expected: [] +metadata_changes: {} +document_overrides: {} diff --git a/conformance/fixtures-unified-v2/families/kimi_k3/dynamo_v2-0.7.15.yaml b/conformance/fixtures-unified-v2/families/kimi_k3/dynamo_v2-0.7.15.yaml new file mode 100644 index 000000000..abfe84c9c --- /dev/null +++ b/conformance/fixtures-unified-v2/families/kimi_k3/dynamo_v2-0.7.15.yaml @@ -0,0 +1,38 @@ +provenance: + status: captured + origin: + crate_version: 0.7.15 + git_commit: 0a4c960f217178d7903c07761c366fbf7439575a + source_sha256: 8083be6e32dae8e8e337dca363273bf1ad96f5b705f3685a331e468df9292d2d +document: + mode: unified +changes: + malformed_json_then_two_valid_calls: + observation: + value: + assembled: + - kind: tool_call + name: echo + arguments: + value: é + - kind: tool_call + name: echo + arguments: + value: Café + chunks: + - expected: [] + - expected: + - kind: tool_call + name: echo + arguments: '{"value":"é"}' + complete: true + - kind: tool_call + name: echo + arguments: '{"value":"Café"}' + complete: true + case_key: UNIFIED.5-4 + stimulus: + ref: current + semantic_sha256: 0787ab8c2229e3c2d6391cdb11e439f4e66a0f0fb22982c92550526404a2caee +metadata_changes: {} +document_overrides: {} diff --git a/conformance/fixtures-unified-v2/families/kimi_k3/inputs_and_golden.yaml b/conformance/fixtures-unified-v2/families/kimi_k3/inputs_and_golden.yaml index 751795eed..93692b2f1 100644 --- a/conformance/fixtures-unified-v2/families/kimi_k3/inputs_and_golden.yaml +++ b/conformance/fixtures-unified-v2/families/kimi_k3/inputs_and_golden.yaml @@ -8483,3 +8483,48 @@ cases: name: get_weather arguments: city: 'null' + malformed_json_then_two_valid_calls: + lifecycle: active + scenario: malformed_json_then_two_valid_calls + description: 'An unfinished JSON string with explicit call closers precedes echo(value=é) and echo(value=Café). Deliver the entire native input in one push, then finish: marker-aligned delivery masks the Kimi K3 0.7.13 lost-call defect. Preserve family malformed-value fallbacks and both later calls.' + policy: + - I2 + - I7 + - P2 + display_id: UNIFIED.5-4 + historical_ids: [] + request: + input: <|open|>tools<|sep|><|open|>call tool="bad" index="1"<|sep|><|open|>json type="object"<|sep|>{"x":"unfinished<|close|>json<|sep|><|close|>call<|sep|><|open|>call tool="echo" index="2"<|sep|><|open|>argument key="value" type="string"<|sep|>é<|close|>argument<|sep|><|close|>call<|sep|><|open|>call tool="echo" index="3"<|sep|><|open|>argument key="value" type="string"<|sep|>Café<|close|>argument<|sep|><|close|>call<|sep|><|close|>tools<|sep|> + init: + starting_state: None + tool_output_mode: Native + named_tool: null + finish_reason: stop + tools: + - name: bad + parameters: + type: object + properties: + value: + type: object + - name: echo + parameters: + type: object + properties: + value: + type: string + required: + - value + chunks: + - delta_text: <|open|>tools<|sep|><|open|>call tool="bad" index="1"<|sep|><|open|>json type="object"<|sep|>{"x":"unfinished<|close|>json<|sep|><|close|>call<|sep|><|open|>call tool="echo" index="2"<|sep|><|open|>argument key="value" type="string"<|sep|>é<|close|>argument<|sep|><|close|>call<|sep|><|open|>call tool="echo" index="3"<|sep|><|open|>argument key="value" type="string"<|sep|>Café<|close|>argument<|sep|><|close|>call<|sep|><|close|>tools<|sep|> + - delta_text: ‹finish› + golden: + assembled: + - kind: tool_call + name: echo + arguments: + value: é + - kind: tool_call + name: echo + arguments: + value: Café diff --git a/conformance/fixtures-unified-v2/families/muse_glimmer/dynamo_v2-0.7.13.yaml b/conformance/fixtures-unified-v2/families/muse_glimmer/dynamo_v2-0.7.13.yaml new file mode 100644 index 000000000..cfefb60f7 --- /dev/null +++ b/conformance/fixtures-unified-v2/families/muse_glimmer/dynamo_v2-0.7.13.yaml @@ -0,0 +1,46 @@ +provenance: + status: captured + origin: + crate_version: 0.7.13 + source_sha256: f1536c9a9fd7ef3999bfb54e7f3e4d316d5df5c558d43ef9959f1668060f2cf8 + git_commit: 47ef952237deefe0643f94025272a558f249687d +document: + mode: unified +changes: + malformed_json_then_two_valid_calls: + case_key: UNIFIED.5-4 + stimulus: + ref: current + semantic_sha256: 6fc5ae94578218d433a32cfefaaf22bc1a65665b45da5c28bcb0338ce615f5a7 + observation: + value: + assembled: + - kind: tool_call + name: bad + arguments: + value: '{"x":"unfinished' + - kind: tool_call + name: echo + arguments: + value: é + - kind: tool_call + name: echo + arguments: + value: Café + chunks: + - expected: + - kind: tool_call + name: bad + arguments: '{"value":"{\"x\":\"unfinished"}' + complete: true + - kind: tool_call + name: echo + arguments: '{"value":"é"}' + complete: true + - kind: tool_call + name: echo + arguments: '{"value":"Café"}' + complete: true + - expected: [] +metadata_changes: {} +document_overrides: {} diff --git a/conformance/fixtures-unified-v2/families/muse_glimmer/inputs_and_golden.yaml b/conformance/fixtures-unified-v2/families/muse_glimmer/inputs_and_golden.yaml index 66273d9d1..4cdfa5a39 100644 --- a/conformance/fixtures-unified-v2/families/muse_glimmer/inputs_and_golden.yaml +++ b/conformance/fixtures-unified-v2/families/muse_glimmer/inputs_and_golden.yaml @@ -8641,3 +8641,100 @@ cases: name: get_weather arguments: city: 'null' + malformed_json_then_two_valid_calls: + lifecycle: active + scenario: malformed_json_then_two_valid_calls + description: 'An unfinished JSON string with explicit call closers precedes echo(value=é) and echo(value=Café). Deliver the entire native input in one push, then finish: marker-aligned delivery masks the Kimi K3 0.7.13 lost-call defect. Preserve family malformed-value fallbacks and both later calls.' + policy: + - I2 + - I7 + - P2 + display_id: UNIFIED.5-4 + historical_ids: [] + request: + input: '<|start|>assistant to=bad<|message|> + + + + {"x":"unfinished + + + + <|eom|><|start|>assistant to=echo<|message|> + + + + é + + + + <|eom|><|start|>assistant to=echo<|message|> + + + + Café + + + + <|eom|>' + init: + starting_state: None + tool_output_mode: Native + named_tool: null + finish_reason: stop + tools: + - name: bad + parameters: + type: object + properties: + value: + type: object + - name: echo + parameters: + type: object + properties: + value: + type: string + required: + - value + chunks: + - delta_text: '<|start|>assistant to=bad<|message|> + + + + {"x":"unfinished + + + + <|eom|><|start|>assistant to=echo<|message|> + + + + é + + + + <|eom|><|start|>assistant to=echo<|message|> + + + + Café + + + + <|eom|>' + - delta_text: ‹finish› + golden: + assembled: + - kind: tool_call + name: bad + arguments: + value: '{"x":"unfinished' + - kind: tool_call + name: echo + arguments: + value: é + - kind: tool_call + name: echo + arguments: + value: Café diff --git a/conformance/fixtures-unified-v2/families/qwen3/dynamo_v2-0.7.13.yaml b/conformance/fixtures-unified-v2/families/qwen3/dynamo_v2-0.7.13.yaml new file mode 100644 index 000000000..275d814f5 --- /dev/null +++ b/conformance/fixtures-unified-v2/families/qwen3/dynamo_v2-0.7.13.yaml @@ -0,0 +1,46 @@ +provenance: + status: captured + origin: + crate_version: 0.7.13 + source_sha256: f1536c9a9fd7ef3999bfb54e7f3e4d316d5df5c558d43ef9959f1668060f2cf8 + git_commit: 47ef952237deefe0643f94025272a558f249687d +document: + mode: unified +changes: + malformed_json_then_two_valid_calls: + case_key: UNIFIED.5-4 + stimulus: + ref: current + semantic_sha256: c630a1238b2c6452f7c475c9e3e2e1fde464a2525db8f5ad1869440c9e34d1d9 + observation: + value: + assembled: + - kind: tool_call + name: bad + arguments: + value: '{"x":"unfinished' + - kind: tool_call + name: echo + arguments: + value: é + - kind: tool_call + name: echo + arguments: + value: Café + chunks: + - expected: + - kind: tool_call + name: bad + arguments: '{"value":"{\"x\":\"unfinished"}' + complete: true + - kind: tool_call + name: echo + arguments: '{"value":"é"}' + complete: true + - kind: tool_call + name: echo + arguments: '{"value":"Café"}' + complete: true + - expected: [] +metadata_changes: {} +document_overrides: {} diff --git a/conformance/fixtures-unified-v2/families/qwen3/inputs_and_golden.yaml b/conformance/fixtures-unified-v2/families/qwen3/inputs_and_golden.yaml index e82488209..4a2d2025d 100644 --- a/conformance/fixtures-unified-v2/families/qwen3/inputs_and_golden.yaml +++ b/conformance/fixtures-unified-v2/families/qwen3/inputs_and_golden.yaml @@ -8311,3 +8311,124 @@ cases: name: get_weather arguments: city: 'null' + malformed_json_then_two_valid_calls: + lifecycle: active + scenario: malformed_json_then_two_valid_calls + description: 'An unfinished JSON string with explicit call closers precedes echo(value=é) and echo(value=Café). Deliver the entire native input in one push, then finish: marker-aligned delivery masks the Kimi K3 0.7.13 lost-call defect. Preserve family malformed-value fallbacks and both later calls.' + policy: + - I2 + - I7 + - P2 + display_id: UNIFIED.5-4 + historical_ids: [] + request: + input: ' + + + + + + {"x":"unfinished + + + + + + + + + + + + é + + + + + + + + + + + + Café + + + + + + ' + init: + starting_state: None + tool_output_mode: Native + named_tool: null + finish_reason: stop + tools: + - name: bad + parameters: + type: object + properties: + value: + type: object + - name: echo + parameters: + type: object + properties: + value: + type: string + required: + - value + chunks: + - delta_text: ' + + + + + + {"x":"unfinished + + + + + + + + + + + + é + + + + + + + + + + + + Café + + + + + + ' + - delta_text: ‹finish› + golden: + assembled: + - kind: tool_call + name: bad + arguments: + value: '{"x":"unfinished' + - kind: tool_call + name: echo + arguments: + value: é + - kind: tool_call + name: echo + arguments: + value: Café diff --git a/conformance/tests/capture_cross_version.rs b/conformance/tests/capture_cross_version.rs index 9705e0a05..172ca26f8 100644 --- a/conformance/tests/capture_cross_version.rs +++ b/conformance/tests/capture_cross_version.rs @@ -43,29 +43,15 @@ use std::path::{Path, PathBuf}; use dynamo_parsers::{ReasoningParser, ReasoningParserType}; use dynamo_parsers_v2::create_tool_parser_for_family; #[cfg(not(conformance_split_only))] -use dynamo_parsers_v2::{ - UnifiedEvent, UnifiedParserExt, assemble, create_unified_parser_for_family, -}; +use dynamo_parsers_v2::{UnifiedEvent, assemble, create_unified_parser_for_family}; use serde_json::{Value, json}; mod common; +use common::unified_capture::{self as capture, CaptureFailure, capture_step, chunk_input}; use common::unified_tools as tools; type Captured = (Vec>, Vec); -#[derive(Debug)] -enum CaptureFailure { - Unavailable(String), - Error(String), -} - -fn capture_step( - result: Result, - stage: &str, -) -> Result { - result.map_err(|error| CaptureFailure::Error(format!("{stage}: {error:#}"))) -} - fn capture_record( result: Result, init: &common::Init, @@ -180,7 +166,8 @@ fn capture_producer_records_init_and_identity() { "guided": {"input": "{\"city\":\"Paris\"}", "init": { "tool_output_mode": "GuidedJson", "named_tool": "get_weather" }}, - "reasoning": {"input": "hiddenvisible", "init": {"starting_state": "Reasoning"}}, + "reasoning": {"input": "hiddenvisible", "init": {"starting_state": "Reasoning"}, + "chunks": [{"delta_text":"hiddenvisible"}, {"delta_text":""}, {"delta_text":"‹finish›"}]}, "native": {"input": "plain response"} }}); std::fs::write( @@ -238,6 +225,13 @@ fn capture_producer_records_init_and_identity() { case["capture_input"]["input"], fixture["cases"][key]["input"] ); + if let Some(requested) = fixture["cases"][key].get("chunks") { + assert_eq!(&case["capture_input"]["chunks"], requested); + assert_eq!( + case["chunks"].as_array().unwrap().len(), + requested.as_array().unwrap().len() + ); + } let expected_tools = fixture["cases"][key] .get("tools") .cloned() @@ -343,7 +337,7 @@ fn native_parser_errors_reach_capture_records() { let mut parser: Box = Box::new(FailingParser { on_finish }); let init = common::Init::default(); - let record = capture_record(native_capture(&mut parser, "text", &init), &init); + let record = capture_record(native_capture(&mut parser, &["text".into()], &init), &init); assert_eq!( record["error"].as_str().unwrap(), if on_finish { @@ -521,37 +515,6 @@ fn split_path_chunks_with_parsers( Ok(rows) } -/// Marker-aligned chunking, byte-for-byte `unified_render::chunk_input`. The per-chunk -/// rows are compared across columns, so a different split would surface as a parser -/// difference that is really a harness difference. -fn chunk_input(input: &str) -> Vec { - let bytes = input.as_bytes(); - let mut chunks = Vec::new(); - let mut i = 0; - let mut text_start = 0; - while i < bytes.len() { - if bytes[i] == b'<' { - if text_start < i { - chunks.push(input[text_start..i].to_string()); - } - let mut j = i + 1; - while j < bytes.len() && bytes[j] != b'>' { - j += 1; - } - let end = (j + 1).min(bytes.len()); - chunks.push(input[i..end].to_string()); - i = end; - text_start = i; - } else { - i += 1; - } - } - if text_start < bytes.len() { - chunks.push(input[text_start..].to_string()); - } - chunks -} - #[cfg(not(conformance_split_only))] fn ev_to_yaml(ev: &UnifiedEvent) -> serde_yaml::Value { serde_yaml::to_value(ev).expect("event serializes") @@ -671,28 +634,23 @@ fn split_path_capture_with_parsers( #[cfg(not(conformance_split_only))] fn native_capture( parser: &mut Box, - input: &str, + chunks: &[String], init: &common::Init, ) -> Result { apply_init(parser, init)?; - let mut deltas = Vec::new(); - let mut rows = Vec::new(); - for ch in chunk_input(input) { - let chunk = capture_step(parser.push(&ch), "native push")?; - rows.push(chunk.iter().map(delta_to_yaml).collect()); - deltas.extend(chunk); - } - let tail = capture_step(parser.finish(), "native finish")?; - if !tail.is_empty() { - rows.push(tail.iter().map(delta_to_yaml).collect()); - deltas.extend(tail); - } + let captured = capture::native_capture(parser, chunks)?; + let deltas: Vec<_> = captured.iter().flatten().cloned().collect(); + let rows = captured + .iter() + .map(|row| row.iter().map(delta_to_yaml).collect()) + .collect(); Ok((rows, assemble(&deltas).iter().map(ev_to_yaml).collect())) } -fn capture_case( +fn capture_case_with_chunks( family: &str, input: &str, + chunks: &[String], init: &common::Init, tool_schemas: &[dynamo_parsers_v2::Tool], ) -> Result { @@ -704,7 +662,7 @@ fn capture_case( "this build has no request initialization API".into(), )); } - return native_capture(&mut parser, input, init); + return native_capture(&mut parser, chunks, init); } Err(error) => format!("{error:#}"), }; @@ -718,6 +676,11 @@ fn capture_case( "split capture cannot apply the requested initialization".into(), )); } + assert_eq!( + chunks, + chunk_input(input), + "legacy split capture cannot replay this schedule" + ); let (rows, assembled) = split_path_capture_with_parsers(&reasoning, &tool, input, tool_schemas)?; let rows = rows @@ -783,7 +746,7 @@ fn capture_rejects_missing_or_malformed_input_fields() { ( 6, "cases: {probe: {input: text, chunks: []}}", - "requested chunks", + "recorded schedule requires finish", ), ( 7, @@ -881,9 +844,15 @@ fn capture_this_build_against_the_current_corpus() { .map(|value| serde_json::to_value(value).expect("case tools schema JSON")); let tool_schema_json = common::unified_tool_schemas_for_case(tool_schema_json.as_ref()); - let mut record = - capture_record(capture_case(&family, input, &init, &tool_schemas), &init); - let chunks: Vec<_> = chunk_input(input) + let recorded: Option> = cdoc + .get("chunks") + .map(|value| serde_yaml::from_value(value.clone()).expect("requested chunks")); + let schedule = capture::replay_chunks(input, recorded.as_deref()); + let mut record = capture_record( + capture_case_with_chunks(&family, input, &schedule, &init, &tool_schemas), + &init, + ); + let chunks: Vec<_> = schedule .into_iter() .chain(std::iter::once("‹finish›".to_string())) .map(|text| serde_json::json!({"delta_text":text})) @@ -897,18 +866,6 @@ fn capture_this_build_against_the_current_corpus() { }, "finish_reason":finish_reason, "tools":tool_schema_json, "chunks":chunks, }); - // This driver computes its delivery schedule. Refuse metadata that - // would claim it executed different tools or chunks. The completion - // reason is request metadata: this API's finish() takes no reason. - for field in ["chunks"] { - if let Some(requested) = cdoc.get(field) { - assert_eq!( - serde_json::to_value(requested).unwrap(), - stimulus[field], - "capture driver cannot apply requested {field}" - ); - } - } record.as_mapping_mut().unwrap().insert( "capture_input".into(), serde_yaml::to_value(stimulus).unwrap(), diff --git a/conformance/tests/common/mod.rs b/conformance/tests/common/mod.rs index 3967e3dcd..8b47c74ab 100644 --- a/conformance/tests/common/mod.rs +++ b/conformance/tests/common/mod.rs @@ -13,6 +13,7 @@ pub mod known_toolcalling_chunking; pub mod known_unified_divergences; +pub mod unified_capture; use std::io::Write; use std::path::{Path, PathBuf}; diff --git a/conformance/tests/common/unified_capture.rs b/conformance/tests/common/unified_capture.rs new file mode 100644 index 000000000..03d05da1a --- /dev/null +++ b/conformance/tests/common/unified_capture.rs @@ -0,0 +1,125 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use serde::Deserialize; + +#[derive(Debug)] +pub enum CaptureFailure { + Unavailable(String), + Error(String), +} + +pub fn capture_step( + result: Result, + stage: &str, +) -> Result { + result.map_err(|error| CaptureFailure::Error(format!("{stage}: {error:#}"))) +} + +/// Tokenize an input into streaming chunks: each control marker (`<...>`, incl. +/// `<|...|>` / `<|"|>`) is its own chunk, and each run of text between markers is +/// a chunk. Generic across the gemma4 / qwen3 / kimi grammars. +pub fn chunk_input(input: &str) -> Vec { + let bytes = input.as_bytes(); + let mut chunks = Vec::new(); + let mut i = 0; + let mut text_start = 0; + while i < bytes.len() { + if bytes[i] == b'<' { + // flush any pending text run + if text_start < i { + chunks.push(input[text_start..i].to_string()); + } + // consume through the matching '>' + let mut j = i + 1; + while j < bytes.len() && bytes[j] != b'>' { + j += 1; + } + let end = (j + 1).min(bytes.len()); + chunks.push(input[i..end].to_string()); + i = end; + text_start = i; + } else { + i += 1; + } + } + if text_start < bytes.len() { + chunks.push(input[text_start..].to_string()); + } + chunks +} + +pub fn input_chunks(input: &str, authored: Option<&[String]>) -> Vec { + match authored { + Some(chunks) => { + assert_eq!( + chunks.concat(), + input, + "authored input_chunks must reproduce input" + ); + chunks.to_vec() + } + None => chunk_input(input), + } +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +pub struct InputChunk { + pub delta_text: String, +} + +pub fn replay_chunks(input: &str, recorded: Option<&[InputChunk]>) -> Vec { + match recorded { + None => input_chunks(input, None), + Some(rows) => { + let (finish, rows) = rows + .split_last() + .expect("recorded schedule requires finish"); + assert_eq!( + finish.delta_text, "‹finish›", + "recorded schedule requires finish" + ); + let chunks: Vec<_> = rows.iter().map(|row| row.delta_text.clone()).collect(); + input_chunks(input, Some(&chunks)) + } + } +} + +#[cfg(not(conformance_split_only))] +pub fn native_capture( + parser: &mut Box, + chunks: &[String], +) -> Result>, CaptureFailure> { + use dynamo_parsers_v2::UnifiedParserExt; + let mut rows = Vec::new(); + for chunk in chunks { + rows.push(capture_step(parser.push(chunk), "native push")?); + } + rows.push( + capture_step(parser.finish(), "native finish")? + .into_iter() + .collect(), + ); + Ok(rows) +} + +#[test] +fn authored_and_recorded_schedules_preserve_utf8_and_empty_chunks() { + let input = "éCafé"; + let authored = vec![input.to_string(), String::new()]; + assert_eq!(input_chunks(input, Some(&authored)), authored); + let mut rows: Vec<_> = authored + .iter() + .map(|s| InputChunk { + delta_text: s.clone(), + }) + .collect(); + rows.push(InputChunk { + delta_text: "‹finish›".into(), + }); + assert_eq!(replay_chunks(input, Some(&rows)), authored); + assert_eq!(input_chunks(input, None), chunk_input(input)); + assert!(std::panic::catch_unwind(|| input_chunks(input, Some(&["wrong".into()]))).is_err()); + assert!(std::panic::catch_unwind(|| replay_chunks(input, Some(&rows[..1]))).is_err()); +} diff --git a/conformance/tests/unified_parity.rs b/conformance/tests/unified_parity.rs index 70f579a73..398a7f699 100644 --- a/conformance/tests/unified_parity.rs +++ b/conformance/tests/unified_parity.rs @@ -16,11 +16,11 @@ //! * `I4` per-stream isolation — two concurrent parsers do not see each other mod common; +use common::unified_capture::{self as capture, chunk_input as chunk_markers}; use common::{ Init, known_unified_divergences::{self as divergences, Check, Expected}, - unified_tools as tools, }; use std::collections::BTreeMap; @@ -42,6 +42,8 @@ struct GoldenFile { struct GoldenCase { input: String, #[serde(default)] + input_chunks: Option>, + #[serde(default)] tools: Option, golden: Vec, /// Request-scoped parser configuration, declared by the case. Shared with @@ -77,20 +79,23 @@ fn events( chunks: &[String], init: &Init, tool_schemas: &[dynamo_parsers_v2::Tool], -) -> Vec { +) -> Result, String> { let mut parser = create_unified_parser_for_family(family, tool_schemas) .unwrap_or_else(|e| panic!("create unified parser for `{family}`: {e}")); init.apply(&mut parser, family); - let mut deltas = Vec::new(); - for chunk in chunks { - deltas.extend(parser.push(chunk).unwrap_or_else(|e| panic!("push: {e}"))); - } - deltas.extend(parser.finish().unwrap_or_else(|e| panic!("finish: {e}"))); - assemble(&deltas) + let rows = capture::native_capture(&mut parser, chunks).map_err(|failure| match failure { + capture::CaptureFailure::Error(error) => error, + other => panic!("unexpected capture failure: {other:?}"), + })?; + Ok(assemble(&rows.into_iter().flatten().collect::>())) } -fn render(events: &[UnifiedEvent]) -> String { +fn render(result: &Result, String>) -> String { + let events = match result { + Ok(events) => events, + Err(error) => return format!("ERROR: {error}"), + }; events .iter() .map(|e| match e { @@ -102,34 +107,6 @@ fn render(events: &[UnifiedEvent]) -> String { .join(" | ") } -/// Marker-aligned chunking: each `<...>` control marker is its own chunk. -fn chunk_markers(input: &str) -> Vec { - let bytes = input.as_bytes(); - let mut chunks = Vec::new(); - let (mut i, mut text_start) = (0, 0); - while i < bytes.len() { - if bytes[i] == b'<' { - if text_start < i { - chunks.push(input[text_start..i].to_string()); - } - let mut j = i + 1; - while j < bytes.len() && bytes[j] != b'>' { - j += 1; - } - let end = (j + 1).min(bytes.len()); - chunks.push(input[i..end].to_string()); - i = end; - text_start = i; - } else { - i += 1; - } - } - if text_start < bytes.len() { - chunks.push(input[text_start..].to_string()); - } - chunks -} - /// Split into chunks of at most `n` chars (never mid-char). fn chunk_every(input: &str, n: usize) -> Vec { input @@ -173,11 +150,11 @@ fn unified_parser_matches_the_golden_oracle() { let case_tools = common::unified_tools_for_schemas(case.tools.as_ref()); let got = events( &file.family, - &chunk_markers(&case.input), + &capture::input_chunks(&case.input, case.input_chunks.as_deref()), &case.init, &case_tools, ); - if got != case.golden { + if got.as_ref() != Ok(&case.golden) { match divergences::expected(&known, &file.family, id, Check::Golden) { Some(Expected::Golden(expected)) if render(&got) == expected.actual => { observed.insert((file.family.clone(), id.clone())); @@ -190,7 +167,7 @@ fn unified_parser_matches_the_golden_oracle() { _ => failures.push(format!( "{id}\n input: {:?}\n golden: {}\n unified: {}", case.input, - render(&case.golden), + render(&Ok(case.golden.clone())), render(&got), )), } @@ -287,7 +264,7 @@ fn unified_parser_has_stream_batch_parity() { let case_tools = common::unified_tools_for_schemas(case.tools.as_ref()); let streamed = events( &file.family, - &chunk_markers(&case.input), + &capture::input_chunks(&case.input, case.input_chunks.as_deref()), &case.init, &case_tools, ); @@ -296,7 +273,7 @@ fn unified_parser_has_stream_batch_parity() { let batch = parser .parse_complete(&case.input) - .unwrap_or_else(|e| panic!("{id}: parse_complete: {e}")); + .map_err(|e| format!("native push: {e:#}")); if batch != streamed { match divergences::expected(&known, &file.family, id, Check::StreamBatch) { Some(Expected::StreamBatch(expected)) @@ -334,6 +311,28 @@ fn unified_parser_has_stream_batch_parity() { ); } +fn advance( + parser: &mut Box, + chunk: Option<&str>, + state: &mut Result, String>, +) { + if let Ok(deltas) = state { + let result = match chunk { + Some(chunk) => parser + .push(chunk) + .map_err(|e| format!("native push: {e:#}")), + None => parser + .finish() + .map(|out| out.events) + .map_err(|e| format!("native finish: {e:#}")), + }; + match result { + Ok(events) => deltas.extend(events), + Err(error) => *state = Err(error), + } + } +} + /// I4: one parser per stream, so interleaving two streams cannot contaminate /// either one. #[test] @@ -347,7 +346,10 @@ fn unified_parsers_are_isolated_per_stream() { let [(id_a, a), (id_b, b)] = pair else { continue; }; - let (ca, cb) = (chunk_markers(&a.input), chunk_markers(&b.input)); + let (ca, cb) = ( + capture::input_chunks(&a.input, a.input_chunks.as_deref()), + capture::input_chunks(&b.input, b.input_chunks.as_deref()), + ); let tools_a = common::unified_tools_for_schemas(a.tools.as_ref()); let tools_b = common::unified_tools_for_schemas(b.tools.as_ref()); let solo_a = events(&file.family, &ca, &a.init, &tools_a); @@ -358,25 +360,25 @@ fn unified_parsers_are_isolated_per_stream() { a.init.apply(&mut pa, id_a); b.init.apply(&mut pb, id_b); - let (mut da, mut db) = (Vec::new(), Vec::new()); + let (mut da, mut db) = (Ok(Vec::new()), Ok(Vec::new())); for i in 0..ca.len().max(cb.len()) { if let Some(c) = ca.get(i) { - da.extend(pa.push(c).unwrap()); + advance(&mut pa, Some(c), &mut da); } if let Some(c) = cb.get(i) { - db.extend(pb.push(c).unwrap()); + advance(&mut pb, Some(c), &mut db); } } - da.extend(pa.finish().unwrap().events); - db.extend(pb.finish().unwrap().events); + advance(&mut pa, None, &mut da); + advance(&mut pb, None, &mut db); assert_eq!( - assemble(&da), + da.map(|d| assemble(&d)), solo_a, "{id_a}: interleaving with {id_b} changed its events" ); assert_eq!( - assemble(&db), + db.map(|d| assemble(&d)), solo_b, "{id_b}: interleaving with {id_a} changed its events" ); @@ -461,6 +463,7 @@ fn manifest_and_parser_registry_agree_on_native_families() { fn deepseek_tool_adapter_matches_both_native_golden_corpora() { use dynamo_parsers_v2::{ToolParseResult, create_tool_parser_for_family}; + let known = divergences::load(); let mut coverage = BTreeMap::::new(); for file in load_golden() .into_iter() @@ -477,14 +480,27 @@ fn deepseek_tool_adapter_matches_both_native_golden_corpora() { .filter(|event| matches!(event, UnifiedEvent::ToolCall { .. })) .collect(); for (label, chunks) in splittings(&case.input) { - let mut parser = create_tool_parser_for_family("deepseek_v4", &tools()).unwrap(); + let case_tools = common::unified_tools_for_schemas(case.tools.as_ref()); + let mut parser = create_tool_parser_for_family("deepseek_v4", &case_tools).unwrap(); let mut result = ToolParseResult::default(); + let mut failure = None; for chunk in chunks { - result.append( - parser - .push(&chunk) - .unwrap_or_else(|e| panic!("{id} {label}: {e}")), - ); + match parser.push(&chunk) { + Ok(delta) => result.append(delta), + Err(error) => { + failure = Some(format!("ERROR: native push: {error:#}")); + break; + } + } + } + if let Some(error) = failure { + let Some(Expected::Golden(expected_error)) = + divergences::expected(&known, &file.family, &id, Check::Golden) + else { + panic!("{id} {label}: unexpected {error}"); + }; + assert_eq!(error, expected_error.actual, "{id} {label}"); + continue; } result.append( parser diff --git a/conformance/tests/unified_render.rs b/conformance/tests/unified_render.rs index 3b90ed0ab..abbefea94 100644 --- a/conformance/tests/unified_render.rs +++ b/conformance/tests/unified_render.rs @@ -24,12 +24,13 @@ use std::path::PathBuf; use dynamo_parsers::{ReasoningParser, ReasoningParserType}; use dynamo_parsers_v2::{ - UnifiedParserExt, assemble, create_tool_parser_for_family, create_unified_parser_for_family, + assemble, create_tool_parser_for_family, create_unified_parser_for_family, }; use serde::Deserialize; use serde_json::{Value, json}; mod common; +use common::unified_capture::{self as capture, chunk_input}; use common::{ Init, @@ -54,6 +55,8 @@ struct GoldenCase { finish_reason: Option, input: String, #[serde(default)] + input_chunks: Option>, + #[serde(default)] tools: Option, golden: Vec, expect: BTreeMap, @@ -178,23 +181,10 @@ fn dynamo_events_with_tools( init: &Init, tool_schemas: &[dynamo_parsers_v2::Tool], ) -> Vec { - if let Ok(mut parser) = create_unified_parser_for_family(family, tool_schemas) { - init.apply(&mut parser, family); - - let mut deltas = Vec::new(); - for chunk in chunk_input(input) { - deltas.extend( - parser - .push(&chunk) - .unwrap_or_else(|e| panic!("unified push `{family}`: {e}")), - ); - } - deltas.extend( - parser - .finish() - .unwrap_or_else(|e| panic!("unified finish `{family}`: {e}")), - ); - return assemble(&deltas).into_iter().map(Ev::from).collect(); + if create_unified_parser_for_family(family, tool_schemas).is_ok() { + return capture_native_case(family, &chunk_input(input), init, tool_schemas) + .expect("native capture") + .0; } let (reasoning_name, tool_family) = parsers_for(family); @@ -240,37 +230,29 @@ fn dynamo_events_with_tools( out } -/// Tokenize an input into streaming chunks: each control marker (`<...>`, incl. -/// `<|...|>` / `<|"|>`) is its own chunk, and each run of text between markers is -/// a chunk. Generic across the gemma4 / qwen3 / kimi grammars. -fn chunk_input(input: &str) -> Vec { - let bytes = input.as_bytes(); - let mut chunks = Vec::new(); - let mut i = 0; - let mut text_start = 0; - while i < bytes.len() { - if bytes[i] == b'<' { - // flush any pending text run - if text_start < i { - chunks.push(input[text_start..i].to_string()); - } - // consume through the matching '>' - let mut j = i + 1; - while j < bytes.len() && bytes[j] != b'>' { - j += 1; - } - let end = (j + 1).min(bytes.len()); - chunks.push(input[i..end].to_string()); - i = end; - text_start = i; - } else { - i += 1; - } - } - if text_start < bytes.len() { - chunks.push(input[text_start..].to_string()); - } - chunks +fn capture_native_case( + family: &str, + chunks: &[String], + init: &Init, + tool_schemas: &[dynamo_parsers_v2::Tool], +) -> Result<(Vec, Vec), capture::CaptureFailure> { + let mut parser = create_unified_parser_for_family(family, tool_schemas) + .unwrap_or_else(|e| panic!("create native parser {family}: {e}")); + init.apply(&mut parser, family); + let rows = capture::native_capture(&mut parser, chunks)?; + let deltas: Vec<_> = rows.iter().flatten().cloned().collect(); + let assembled = assemble(&deltas).into_iter().map(Ev::from).collect(); + let rows = chunks + .iter() + .cloned() + .chain(std::iter::once("‹finish›".into())) + .zip(rows) + .map(|(delta_text, deltas)| ChunkRow { + delta_text, + deltas: deltas.iter().map(unified_delta_json).collect(), + }) + .collect(); + Ok((assembled, rows)) } /// One streaming chunk: the delta text fed, and the RAW per-chunk deltas Dynamo @@ -302,29 +284,10 @@ fn dynamo_chunks_with_tools( init: &Init, tool_schemas: &[dynamo_parsers_v2::Tool], ) -> Vec { - // Unified families: ONE parser, so a chunk's deltas are simply what it emitted. - if let Ok(mut parser) = create_unified_parser_for_family(family, tool_schemas) { - init.apply(&mut parser, family); - - let mut rows = Vec::new(); - for chunk in chunk_input(input) { - let deltas = parser.push(&chunk).expect("native capture push failed"); - rows.push(ChunkRow { - delta_text: chunk, - deltas: deltas.iter().map(unified_delta_json).collect(), - }); - } - let tail: Vec = parser - .finish() - .expect("native capture finish failed") - .iter() - .map(unified_delta_json) - .collect(); - rows.push(ChunkRow { - delta_text: "‹finish›".to_string(), - deltas: tail, - }); - return rows; + if create_unified_parser_for_family(family, tool_schemas).is_ok() { + return capture_native_case(family, &chunk_input(input), init, tool_schemas) + .expect("native capture") + .1; } let (reasoning_name, tool_family) = parsers_for(family); @@ -594,10 +557,27 @@ fn render_unified_conformance_html() { // Dynamo: live. let case_tools = common::unified_tools_for_schemas(case.tools.as_ref()); - let got = dynamo_events_with_tools(&file.family, &case.input, &case.init, &case_tools); - let dclass = classify(&file.family, &case.golden, &got); + let schedule = capture::input_chunks(&case.input, case.input_chunks.as_deref()); + let (got, captured_rows, error) = + match capture_native_case(&file.family, &schedule, &case.init, &case_tools) { + Ok((got, rows)) => (got, rows, None), + Err(capture::CaptureFailure::Error(error)) => { + (Vec::new(), Vec::new(), Some(error)) + } + Err(failure) => panic!("native capture unavailable: {failure:?}"), + }; + let dclass = if error.is_some() { + "ERROR" + } else { + classify(&file.family, &case.golden, &got) + }; if dclass != "MATCH" { - let actual = got.iter().map(Ev::render).collect::>().join(" | "); + let actual = error + .as_ref() + .map(|e| format!("ERROR: {e}")) + .unwrap_or_else(|| { + got.iter().map(Ev::render).collect::>().join(" | ") + }); match divergences::expected(&known, &file.family, id, Check::Golden) { Some(Expected::Golden(expected)) if actual == expected.actual => { observed_dynamo.insert((file.family.clone(), id.clone())); @@ -621,11 +601,19 @@ fn render_unified_conformance_html() { "{id:44} dynamo={dclass:6} :: {}", got.iter().map(Ev::render).collect::>().join(" | ") ); - let chunk_feed: Vec = - dynamo_chunks_with_tools(&file.family, &case.input, &case.init, &case_tools) + let chunk_feed: Vec = if error.is_some() { + schedule + .iter() + .cloned() + .chain(std::iter::once("‹finish›".into())) + .map(|text| json!({"delta_text": text})) + .collect() + } else { + captured_rows .into_iter() - .map(|r| json!({"delta_text": r.delta_text, "dynamo": r.deltas})) - .collect(); + .map(|r| json!({"delta_text":r.delta_text,"dynamo":r.deltas})) + .collect() + }; let scenario = id .strip_prefix("UNIFIED.") @@ -644,6 +632,7 @@ fn render_unified_conformance_html() { "tools": common::unified_tool_schemas_for_case(case.tools.as_ref()), "golden": case.golden, "dynamo": got, + "dynamo_error": error, "dynamo_verdict": dclass, "vllm_verdict": vx.map(|e| if e.verdict == "match" { "MATCH".to_string() } else { e.class.clone().unwrap_or_else(|| "DIVERGE".into()) }), "vllm_note": vx.and_then(|e| e.note.clone()), @@ -805,6 +794,8 @@ struct CaptureDoc { #[derive(Deserialize)] struct CaptureCase { + #[serde(default)] + error: Option, #[serde(default)] assembled: Vec, #[serde(default)] @@ -838,6 +829,8 @@ struct InputCase { init: Init, #[serde(default)] tools: Option, + #[serde(default)] + chunks: Option>, } /// GUARD: the COMMITTED Dynamo capture must equal what the parsers produce NOW. @@ -897,7 +890,7 @@ fn validate_selected_dynamo_capture(root: &std::path::Path, capture_dir: &std::p // PR-qualified sparse overlays. New cases must carry their input metadata in // the same overlay as the capture, rather than making the released shard mutable. type CaptureKey = (String, String); - type CaptureMetadata = (String, String, Init, Option); + type CaptureMetadata = InputCase; let mut meta: BTreeMap = BTreeMap::new(); for input_dir in input_dirs { for entry in glob_yaml(&input_dir) { @@ -906,13 +899,10 @@ fn validate_selected_dynamo_capture(root: &std::path::Path, capture_dir: &std::p for (key, case) in doc.cases { // A sparse overlay can rename a case while preserving its scenario. // The newer key replaces the released key for capture validation. - meta.retain(|(family, _), (scenario, _, _, _)| { - family != &doc.family || scenario != &case.scenario + meta.retain(|(family, _), previous| { + family != &doc.family || previous.scenario != case.scenario }); - meta.insert( - (doc.family.clone(), key), - (case.scenario, case.input, case.init, case.tools), - ); + meta.insert((doc.family.clone(), key), case); } } } @@ -937,9 +927,7 @@ fn validate_selected_dynamo_capture(root: &std::path::Path, capture_dir: &std::p } for doc in captures { for (key, committed) in doc.cases { - let Some((scenario, input, init, tool_schemas)) = - meta.get(&(doc.family.clone(), key.clone())) - else { + let Some(metadata) = meta.get(&(doc.family.clone(), key.clone())) else { stale.push(format!( "{} [{key}] has no input metadata in inputs or its PR overlays", doc.family @@ -947,10 +935,36 @@ fn validate_selected_dynamo_capture(root: &std::path::Path, capture_dir: &std::p continue; }; checked += 1; + let InputCase { + scenario, + input, + init, + tools: tool_schemas, + chunks, + } = metadata; let id = format!("UNIFIED.{scenario}.{}", doc.family); let case_tools = common::unified_tools_for_schemas(tool_schemas.as_ref()); - let live_assembled = dynamo_events_with_tools(&doc.family, input, init, &case_tools); + let schedule = capture::replay_chunks(input, chunks.as_deref()); + let (live_assembled, live_rows) = + match capture_native_case(&doc.family, &schedule, init, &case_tools) { + Ok(value) => { + if committed.error.is_some() { + stale.push(format!("{id}: recorded error no longer occurs")); + continue; + } + value + } + Err(capture::CaptureFailure::Error(error)) + if committed.error.as_ref() == Some(&error) => + { + continue; + } + Err(failure) => { + stale.push(format!("{id}: unexpected live error {failure:?}")); + continue; + } + }; if live_assembled != committed.assembled { stale.push(format!( "{id} [{key}] assembled\n committed: {}\n live: {}", @@ -970,11 +984,7 @@ fn validate_selected_dynamo_capture(root: &std::path::Path, capture_dir: &std::p } // The page assembles the Dynamo column from these per-chunk deltas, so // they have to be current too — not just the assembled list. - let live_chunks: Vec> = - dynamo_chunks_with_tools(&doc.family, input, init, &case_tools) - .into_iter() - .map(|r| r.deltas) - .collect(); + let live_chunks: Vec> = live_rows.into_iter().map(|r| r.deltas).collect(); let committed_chunks: Vec> = committed.chunks.into_iter().map(|c| c.expected).collect(); if live_chunks != committed_chunks { diff --git a/conformance/tests/unified_schema_roundtrip.rs b/conformance/tests/unified_schema_roundtrip.rs index 40dedd87f..4ad684d37 100644 --- a/conformance/tests/unified_schema_roundtrip.rs +++ b/conformance/tests/unified_schema_roundtrip.rs @@ -39,6 +39,8 @@ struct GoldenCase { finish_reason: Option, /// Raw streamed model text. input: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + input_chunks: Option>, /// Per-case request schema for cases whose argument typing differs from the shared corpus schema. #[serde(default, skip_serializing_if = "Option::is_none")] tools: Option, @@ -128,6 +130,7 @@ fn every_case_is_well_formed() { file.family ); assert!(!case.input.is_empty(), "{path}: `{id}` has empty input"); + common::unified_capture::input_chunks(&case.input, case.input_chunks.as_deref()); if let Some(tools) = &case.tools { assert!(tools.is_array(), "{path}: `{id}` tools must be an array"); } diff --git a/conformance/unified-known-divergences.yaml b/conformance/unified-known-divergences.yaml index a1d9fb41a..ad0fac085 100644 --- a/conformance/unified-known-divergences.yaml +++ b/conformance/unified-known-divergences.yaml @@ -6,4 +6,13 @@ # fails CI. Entries that stop diverging fail as stale and must be removed. # Schema: family -> case ID -> check -> note and exact observed output. -{} +gemma4: + UNIFIED.malformed_json_then_two_valid_calls.gemma4: + golden: + note: An unfinished native string consumes both later echo calls; parser fix is outside this capture change. + actual: '' +deepseek_v41: + UNIFIED.malformed_json_then_two_valid_calls.deepseek_v41: + golden: + note: Malformed object parameter raises before either later echo call; parser fix is outside this capture change. + actual: 'ERROR: native push: EOF while parsing a string at line 1 column 16' diff --git a/conformance/utils/lib/parsers/UNIFIED_CASES.md b/conformance/utils/lib/parsers/UNIFIED_CASES.md index 626e30e00..1a4e72e27 100644 --- a/conformance/utils/lib/parsers/UNIFIED_CASES.md +++ b/conformance/utils/lib/parsers/UNIFIED_CASES.md @@ -82,6 +82,7 @@ New case IDs always use a numeric suffix: `-` for numeric groups or `< - **`5-1`** (`truncated_tool_eof`) EOF mid-call. Golden drops the partial, keeps preceding output (P2); vLLM Rust hard-errors (`ParsingFailed`). Class ERROR. - **`5-2`** (`tool_no_close`) Complete call body but the close marker never arrives. Most grammars recover the complete call at finish; DeepSeek V4 and V4.1 require the invoke closer and drop this malformed call. This is also covered in: TOOLCALLING.streamv1.5.a. - **`5-3`** (`orphan_close_after_prose`) Orphan close marker after prose. Golden strips it; engines may leak. Class LEAK. +- **`5-4`** (`malformed_json_then_two_valid_calls`) Across all eight families, an unfinished native JSON/string body with explicit call closers precedes `echo(value="é")` and `echo(value="Café")`. The authored `input_chunks: [input]` delivers one `push()`, then `finish()`. Marker-aligned delivery masks the Kimi K3 defect: Published `dynamo-parsers-v2 = 0.7.13` and `0.7.14` lose the first echo, and unpublished candidate `0.7.15` preserves both. The candidate moved to `0.7.15` after `0.7.14` was published; its earlier unpublished checkpoint is retired rather than relabeled as released output. K2 retains the malformed call as `{}`; Qwen3, GLM47, DeepSeek V4, and Muse retain its malformed object value as a literal string. Those five families preserve both echoes in both versions. Gemma4 returns no calls and DeepSeek V4.1 raises a JSON parse error in both versions; their goldens still require both echoes, and both remain red. Parser fixes for these two failures are outside this change. ### Group 6 — TC Empty body (TOOLCALLING.streamv1.6) - **`6-1`** (`empty_args`) Call with `{}` arguments. Must emit the call with an empty object, not drop it. This is also covered in: TOOLCALLING.streamv1.6.a. @@ -132,7 +133,7 @@ New case IDs always use a numeric suffix: `-` for numeric groups or `< - **`12-4`** (`tool_in_reason_with_text`) 12-2 WITH visible narration before and after — text → reason → call → reason → text. Golden breaks out and keeps the surrounding text; engines leak the nested markup. Class LEAK. ### DeepSeek V4.1 applicability -- DeepSeek V4.1 uses the ordered Unified contract for native DSML calls, reasoning interleaving, guided JSON, and prefilled states. The current corpus emits 93 of the 112 taxonomy cases for this family. +- DeepSeek V4.1 uses the ordered Unified contract for native DSML calls, reasoning interleaving, guided JSON, and prefilled states. The current corpus emits 94 of the 113 taxonomy cases for this family. - Every taxonomy scenario declared for DeepSeek V4.1 is generated. The applicable cases include `30-13`; the Guided Decoding groups `31-1` through `35-2` except `muse-1`; the marker-discriminating Response row `50-4`; and `40-1` through `40-4` plus `41-1` through `41-2`. The native prefilled cases `40-1`, `40-3`, and `40-4` retain explicit inputs and outputs even though other DSv4.1 rows exercise the same transitions. - The 19 omitted cases are `kimi-1` through `kimi-8`, which require Kimi K3 XTML syntax; `gemma-1` through `gemma-2`, which require Gemma 4 guided call-prefix syntax; `glm5-1`, which requires GLM's argument-marker grammar; `7-4.mixed_labels`, which reproduces GLM's mixed nullable fields; `7-4.ref`, `7-5.ref`, `7-9`, `7-11`, `7-12`, and `7-13`, which probe GLM schema references; `muse-1`, whose non-Muse variant duplicates `35-1`. The `muse-1` duplicate does not imply that quoted or malformed model output cannot occur. - `30-13` retains the historical bare header with no tool name. `34-1` uses an unfinished DSML invoke header inside reasoning rather than a completed calls-block opener. Marker-free prefilled-Response rows are omitted because their default-state siblings already cover native and guided valid, multi-call, truncated, and malformed inputs; `50-4` proves that Response treats reasoning markers as visible text. @@ -345,6 +346,8 @@ The marker-free prefilled-Response variants were removed because they emitted th ## Authoring a case: what to check BEFORE adding one +Numeric groups describe shared behaviors: inspect every registered family and author its native input and expected output wherever applicable. Named family sections hold grammar-specific cases; explain genuine exclusions. A bug discovered in one family does not justify restricting shared coverage to it. Preserve previously published IDs. + Every rule here exists because a case was added that could not fail for the reason it claimed. Fake coverage is worse than no coverage — it renders green. 1. **Is it distinguishable from an existing case?** Compare `init` AND input against the corpus. If some existing case has the same configuration and the same input shape, the new case tests nothing. Three groups were deleted for exactly this: a whole axis whose 51/52/53 cases had the same config and inputs as `1-1`, differing only in a label the parser cannot read. diff --git a/conformance/utils/src/assets/conformance.js b/conformance/utils/src/assets/conformance.js index 5c3fe406b..3727e040f 100644 --- a/conformance/utils/src/assets/conformance.js +++ b/conformance/utils/src/assets/conformance.js @@ -227,8 +227,13 @@ // (a real, non-error output) is a genuine disagreement with the failed Reference -> // red (a delta), not grey n/a. If nothing parsed to disagree with (no Compares, or // every Compare also threw / is unavailable), there is no delta -> neutral grey. + // Oracle-backed cells fail on a Reference error even with no Compare selected. + const redOnDiff = cell.getAttribute('data-red-on-diff') === '1'; if (bd.err === 1) { - if (avail.length > 0) { + if (redOnDiff) { + cell.classList.add('cmp-leak'); if (marker) { marker.textContent = '✗'; } + if (countThis) { counts.problem++; } + } else if (avail.length > 0) { cell.classList.add('cmp-leak'); if (marker) { marker.textContent = String(avail.length) + 'Δ'; } if (countThis) { counts.problem++; } } else { @@ -239,7 +244,6 @@ } const diffs = avail.filter(function (o) { return o.sig !== refSig; }).length; const leak = bd.leak === 1; - const redOnDiff = cell.getAttribute('data-red-on-diff') === '1'; // Unified tab (data-red-on-diff): GOLDEN is the fixed oracle, and the cell color // reflects ONLY the REF — the starred engine (`base`, default Dynamo) — measured // against golden. A red cell means the REF's own output diverges from golden. The diff --git a/conformance/utils/src/capture_stimulus.py b/conformance/utils/src/capture_stimulus.py index 587b6933d..2830b84c3 100644 --- a/conformance/utils/src/capture_stimulus.py +++ b/conformance/utils/src/capture_stimulus.py @@ -172,8 +172,14 @@ def validated_current_capture_docs(directory: Path, input_dirs: list[Path]) -> l failure = comparison_failure(record, current, raw, relative, bindings) if failure: raise ValueError(f"{ident}: {failure}") - if "unavailable" in record or "error" in record: + if "unavailable" in record: raise ValueError(f"current capture did not succeed: {ident}") + if "error" in record: + known = yaml.safe_load((Path(__file__).resolve().parents[2] / "unified-known-divergences.yaml").read_text()) + scenario_id = f"UNIFIED.{current.get('scenario', '')}.{ident[0]}" + expected_error = known.get(ident[0], {}).get(scenario_id, {}).get("golden", {}).get("actual") + if expected_error != f"ERROR: {record['error']}" or "assembled" in record or "chunks" in record: + raise ValueError(f"current capture did not succeed (unexpected error): {ident}") families = {} for (family, key), (record, _raw, _relative, _bindings) in sorted(captures.items()): families.setdefault(family, {})[input_keys[(family, key)]] = record diff --git a/conformance/utils/src/explode_unified_fixtures.py b/conformance/utils/src/explode_unified_fixtures.py index 72dc52cf1..a56330a7a 100644 --- a/conformance/utils/src/explode_unified_fixtures.py +++ b/conformance/utils/src/explode_unified_fixtures.py @@ -152,6 +152,9 @@ def slot(dirname, family, captured_with=None, model_label=None): "chunks": [{"expected": ch.get("dynamo") or []} for ch in chunks], } + if c.get("dynamo_error"): + slot(ddir, fam)[key] = {"capture_input": capture_input(c), "error": c["dynamo_error"]} + # peer engine overlays for impl in ("vllm_python", "vllm_rust", "sglang_python"): res = (caps[impl].get("results") or {}).get(cid) diff --git a/conformance/utils/src/gen_unified_golden.py b/conformance/utils/src/gen_unified_golden.py index 44b88a4e8..70af62dfc 100644 --- a/conformance/utils/src/gen_unified_golden.py +++ b/conformance/utils/src/gen_unified_golden.py @@ -1836,6 +1836,53 @@ def _edge_case_family_policy(edge_case): return edge_case[-2] if len(edge_case) == 8 else edge_case[-1] +RECOVERY_RAW = '{"x":"unfinished' +RECOVERY_TOOLS = [ + {"name": "bad", "parameters": {"type": "object", "properties": {"value": {"type": "object"}}}}, + {"name": "echo", "parameters": {"type": "object", "properties": {"value": {"type": "string"}}, "required": ["value"]}}, +] + + +def malformed_json_recovery(fam): + following = [{"kind": "tool_call", "name": "echo", "arguments": {"value": value}} for value in ("é", "Café")] + if fam == "kimi_k3": + inp = k3_tools(k3_call("bad", 1, k3_json(RECOVERY_RAW)) + "".join( + k3_call("echo", index, k3_argument("value", "string", value)) + for index, value in ((2, "é"), (3, "Café")))) + golden = following + elif fam == "kimi_k2": + inp = ("<|tool_calls_section_begin|><|tool_call_begin|>functions.bad:0" + f"<|tool_call_argument_begin|>{RECOVERY_RAW}<|tool_call_end|><|tool_calls_section_end|>") + golden = [{"kind": "tool_call", "name": "bad", "arguments": {}}] + following + elif fam == "gemma4": + inp = '<|tool_call>call:bad{value:{x:<|"|>unfinished}}' + golden = following + else: + inp = r_tool(fam, "bad", "value", RECOVERY_RAW, 0) + if fam in ("deepseek_v4", "deepseek_v41"): + inp = inp.replace('string="true"', 'string="false"') + golden = ([{"kind": "tool_call", "name": "bad", "arguments": {"value": RECOVERY_RAW}}] + if fam != "deepseek_v41" else []) + following + if fam != "kimi_k3": + inp += "".join(r_tool(fam, "echo", "value", value, index) + for index, value in ((1, "é"), (2, "Café"))) + return inp, golden + + +EDGE.append(( + "malformed_json_then_two_valid_calls", + "An unfinished JSON string with explicit call closers precedes echo(value=é) and echo(value=Café). Deliver the entire native input in one push, then finish: marker-aligned delivery masks the Kimi K3 0.7.13 lost-call defect. Preserve family malformed-value fallbacks and both later calls.", + ["I2", "I7", "P2"], [], + {"starting_state": "None", "tool_output_mode": "Native", "named_tool": None}, + {"finish_reason": "stop", "single_chunk": True}, + {fam: (malformed_json_recovery(fam)[0], D("UNSUPPORTED", "No peer capture recorded."), + D("LOSS" if fam == "gemma4" else "ERROR", "Known malformed-call recovery defect; later calls must survive.") + if fam in ("gemma4", "deepseek_v41") else M, malformed_json_recovery(fam)[1]) + for fam in FAMILIES}, + RECOVERY_TOOLS, +)) + + DEEPSEEK_V41_SCENARIOS = { spec[0] for spec in (*CLEAN, *EDGE) @@ -2034,6 +2081,8 @@ def _deepseek_v41_input(segments): )) + + def build_cases(fam): """Every CLEAN + EDGE scenario for one family, keyed by case id.""" cases = {} @@ -2052,6 +2101,9 @@ def build_cases(fam): "finish_reason": "stop", } cases.update(_build_edge_cases(fam, EDGE)) + if fam in ("gemma4", "deepseek_v41"): + case = cases[f"UNIFIED.malformed_json_then_two_valid_calls.{fam}"] + case["expect"]["dynamo_current"] = case["expect"]["dynamo"] if fam == "deepseek_v41": case = cases[f"UNIFIED.reason_unterminated.{fam}"] case["input"] = case["input"].removeprefix("") @@ -2153,6 +2205,8 @@ def _build_edge_cases(fam, specs): "init": init, "finish_reason": stream_config.get("finish_reason", "stop"), } + if stream_config.get("single_chunk"): + case["input_chunks"] = [inp] if case_tools is not None: case["tools"] = case_tools[fam] if isinstance(case_tools, dict) else case_tools cases[cid] = case @@ -2217,6 +2271,11 @@ def emit_yaml(fam): lines.append(f" {ln}") lines.append(f" golden: {json.dumps(c['golden'], ensure_ascii=False)}") lines.append(f" expect: {json.dumps(c['expect'], ensure_ascii=False)}") + if "input_chunks" in c: + chunks = c["input_chunks"] + if not isinstance(chunks, list) or any(not isinstance(chunk, str) for chunk in chunks) or "".join(chunks) != c["input"]: + raise ValueError(f"{cid}: input_chunks must concatenate to input") + lines.append(f" input_chunks: {json.dumps(chunks, ensure_ascii=False)}") if c.get("tools") is not None: lines.append(f" tools: {json.dumps(c['tools'], ensure_ascii=False)}") return "\n".join(lines) + "\n" diff --git a/conformance/utils/src/generate_conformance_table.py b/conformance/utils/src/generate_conformance_table.py index 7a90e581b..dbebd84e8 100644 --- a/conformance/utils/src/generate_conformance_table.py +++ b/conformance/utils/src/generate_conformance_table.py @@ -2121,7 +2121,7 @@ def _cat(evs, want_reason): return "LOSS" -def _assemble_stream(chunk_deltas: list) -> list: +def _assemble_stream(chunk_deltas: list, *, recorded_assembly: list | None = None) -> list: """Assemble the FINAL ordered event list from a parser's per-chunk STREAMED deltas (not its batch final message). Coalesces consecutive reasoning/text runs and joins per-call tool-argument fragments (a delta with a name starts a new call; nameless @@ -2160,7 +2160,8 @@ def _assemble_stream(chunk_deltas: list) -> list: if cur_tool is not None and dl.get("complete", True): events[cur_tool]["_complete"] = True events = [e for e in events if e.get("kind") != "tool_call" or e.get("_complete", True)] - for e in events: + malformed = set() + for position, e in enumerate(events): if e["kind"] == "tool_call": raw = e.pop("_raw", "") e.pop("_complete", None) @@ -2172,7 +2173,23 @@ def _assemble_stream(chunk_deltas: list) -> list: try: e["arguments"] = json.loads(raw) except (ValueError, TypeError): + malformed.add(position) e["arguments"] = raw + if malformed and recorded_assembly is not None: + calls = [(position, event) for position, event in enumerate(events) if event["kind"] == "tool_call"] + recorded = [event for event in recorded_assembly if event["kind"] == "tool_call"] + # Native P3 uses {}, but split-era captures preserve invalid JSON bytes. + # Require the entire call sequence to align before using recorded fallbacks; + # split batch assembly can differ from stream order and must not replace it. + if len(calls) == len(recorded) and all( + event["name"] == observed["name"] and ( + event["arguments"] == observed["arguments"] or + position in malformed and observed["arguments"] == {} + ) for (position, event), observed in zip(calls, recorded) + ): + for (position, event), observed in zip(calls, recorded): + if position in malformed: + event["arguments"] = observed["arguments"] return events @@ -2679,7 +2696,7 @@ def _unified_na_cell(family: str, scenario: str, group_num: int) -> dict: dyn_chunk_deltas = [ch.get("dynamo") or [] for ch in (c.get("chunks") or [])] if dynamo_failure: dyn_chunk_deltas = [] - dyn = _assemble_stream(dyn_chunk_deltas) + dyn = _assemble_stream(dyn_chunk_deltas, recorded_assembly=c.get("dynamo")) gsig, dsig = _sig(gold), _sig(dyn) dverd = _unified_classify(f, gold, dyn) if dynamo_failure: @@ -2746,7 +2763,7 @@ def _unified_na_cell(family: str, scenario: str, group_num: int) -> dict: ) prev_chunks_by_ver[pv] = [] continue - pevents = _assemble_stream(pchunks) + pevents = _assemble_stream(pchunks, recorded_assembly=pdoc.get("assembled")) pverd = _unified_classify(f, gold, pevents) cmp[f"dynamo@{pv}"] = markers.cmp_entry( _sig(pevents), leak=1 if pverd == "LEAK" else 0 @@ -2827,10 +2844,10 @@ def _unified_na_cell(family: str, scenario: str, group_num: int) -> dict: else f"not captured at {pv} — this case postdates that build")} if (prev_by_ver.get(pv) is None) else (_unified_capture_failure(prev_by_ver[pv]) or { - "events": _assemble_stream(prev_chunks_by_ver.get(pv) or []), + "events": _assemble_stream(prev_chunks_by_ver.get(pv) or [], recorded_assembly=prev_by_ver[pv].get("assembled")), "verdict": _unified_classify( f, gold, - _assemble_stream(prev_chunks_by_ver.get(pv) or [])), + _assemble_stream(prev_chunks_by_ver.get(pv) or [], recorded_assembly=prev_by_ver[pv].get("assembled"))), "explanation": ( f"Inherited unchanged from Dynamo v2 {prev_by_ver[pv].get('inherited_from')}." if prev_by_ver[pv].get("inherited_from") else None diff --git a/conformance/utils/src/unified_taxonomy.py b/conformance/utils/src/unified_taxonomy.py index 475499f49..ea0175dae 100644 --- a/conformance/utils/src/unified_taxonomy.py +++ b/conformance/utils/src/unified_taxonomy.py @@ -40,6 +40,7 @@ # Group 5 — Truncation / recovery (streamv1.5) "truncated_tool_eof": (5, "1"), "tool_no_close": (5, "2"), "orphan_close_after_prose": (5, "3"), + "malformed_json_then_two_valid_calls": (5, "4"), # Group 6 — Empty body (streamv1.6) "empty_args": (6, "1"), # Group 7 — Argument fidelity (streamv1.7) diff --git a/conformance/utils/tests/schema_oracle.py b/conformance/utils/tests/schema_oracle.py index ab7f99bb2..394b9b148 100644 --- a/conformance/utils/tests/schema_oracle.py +++ b/conformance/utils/tests/schema_oracle.py @@ -33,7 +33,7 @@ def matches_schema( ) -> bool: root_schema = schema if root_schema is None else root_schema assert schema.keys() <= {"type", "properties", "items", "anyOf", "oneOf", "const", - "enum", "nullable", "minLength", "$ref", "$defs", "definitions"}, schema + "enum", "nullable", "minLength", "required", "$ref", "$defs", "definitions"}, schema if "$ref" in schema: reference = schema["$ref"] assert reference not in _references, ("cyclic schema reference", reference) @@ -64,6 +64,8 @@ def matches_schema( if isinstance(value, str) and len(value) < schema.get("minLength", 0): return False if isinstance(value, dict): + if not set(schema.get("required", [])) <= value.keys(): + return False properties = schema.get("properties", {}) return all(matches_schema(item, properties[key], root_schema, _references) for key, item in value.items() if key in properties) if isinstance(value, list) and "items" in schema: diff --git a/conformance/utils/tests/test_browser_popup_compare.py b/conformance/utils/tests/test_browser_popup_compare.py index 6d1e63ad5..7076748c1 100644 --- a/conformance/utils/tests/test_browser_popup_compare.py +++ b/conformance/utils/tests/test_browser_popup_compare.py @@ -558,3 +558,32 @@ def test_no_doubled_assembled_call_names_in_dom(driver): """ ) assert not dupes, f"doubled assembled call names in the DOM: {dupes[:3]}" + + +def test_unified_recovery_colors_follow_selected_reference(driver): + driver.execute_script("document.querySelector('[data-tab-target=tab-unified]').click()") + try: + for version in ("dynamo", "dynamo@0.7.13"): + _select(driver, version, []) + cells = driver.execute_script(""" + const model = JSON.parse(document.getElementById('conformance-model').textContent); + const tab = model.tabs.find(t => t.id === 'tab-unified'); + return tab.rows.map(row => { + const expected = row.cells.malformed_json_then_two_valid_calls.cmp; + const cell = [...document.querySelectorAll('#tab-unified td.cell[data-cmp]')] + .find(cell => cell.dataset.family === row.family && + cell.dataset.cmp === JSON.stringify(expected)); + return {family: row.family, classes: cell.className, + marker: cell.querySelector('.marker-text').textContent}; + }); + """) + assert len(cells) == 8 + for cell in cells: + red = cell["family"] in {"gemma4", "deepseek_v41"} or ( + version == "dynamo@0.7.13" and cell["family"] == "kimi_k3") + assert ("cmp-leak" if red else "cmp-eq") in cell["classes"], cell + if red: + assert cell["marker"] == "✗", cell + finally: + _select(driver, "dynamo", []) + _open_stream_tab(driver) diff --git a/conformance/utils/tests/test_capture_stimulus.py b/conformance/utils/tests/test_capture_stimulus.py index d0cce9333..a049519fa 100644 --- a/conformance/utils/tests/test_capture_stimulus.py +++ b/conformance/utils/tests/test_capture_stimulus.py @@ -237,3 +237,16 @@ def test_current_source_uses_only_complete_snapshot_records(tmp_path, selected_p assert capture_stimulus.validated_current_capture_docs(selected, [tmp_path / "inputs"]) == [ {"family": "deepseek_v41", "cases": {"UNIFIED.7-2": record}}, ] + + +def test_recorded_known_error_is_bound_and_must_match_exactly(tmp_path): + current = _input("same") | {"scenario": "malformed_json_then_two_valid_calls"} + error = "native push: EOF while parsing a string at line 1 column 16" + record = {"capture_input": capture_stimulus.capture_input(current), "error": error} + _write(tmp_path, "inputs", current, "UNIFIED.5-4") + _write(tmp_path, "capture", record, "UNIFIED.5-4") + assert capture_stimulus.validate_current_capture(tmp_path / "capture", [tmp_path / "inputs"]) == 1 + for invalid in ({"error": error + " changed"}, {"assembled": []}, {"chunks": []}): + _write(tmp_path, "capture", record | invalid, "UNIFIED.5-4") + with pytest.raises(ValueError, match="unexpected error"): + capture_stimulus.validate_current_capture(tmp_path / "capture", [tmp_path / "inputs"]) diff --git a/conformance/utils/tests/test_model.py b/conformance/utils/tests/test_model.py index a0c341420..d2042c828 100644 --- a/conformance/utils/tests/test_model.py +++ b/conformance/utils/tests/test_model.py @@ -1132,7 +1132,9 @@ def test_glm_type_references_have_typed_current_batch_and_unified_captures( latest = [next(candidate for candidate in batch["candidates"] if candidate["key"].startswith(implementation) and candidate["parse_mode"] == mode) for implementation, mode in (("dynamo_v1", "batch"), ("dynamo_v2", "stream"))] - assert latest[1]["version"] == dynamo_v2_label(REPO) + # The batch tab's stream candidate retains its measured version after a Unified-only refresh. + recorded = _peer_versions("toolcalling/fixtures-stream-v1")["dynamo_v2"] + assert latest[1]["version"] in {version.split("+source.", 1)[0] for version in recorded} for candidate in latest: block = blocks[candidate["key"]] assert block["calls"] == calls diff --git a/conformance/utils/tests/test_unified_stream_assembly.py b/conformance/utils/tests/test_unified_stream_assembly.py index ca4e99e15..1f37aab5b 100644 --- a/conformance/utils/tests/test_unified_stream_assembly.py +++ b/conformance/utils/tests/test_unified_stream_assembly.py @@ -1,3 +1,4 @@ +import copy import importlib.util import sys from pathlib import Path @@ -46,3 +47,51 @@ def test_complete_tool_delta_keeps_prior_argument_fragments(): "name": "get_weather", "arguments": {"city": "Paris"}, }] + + +def test_dynamo_malformed_arguments_use_p3_without_hiding_peer_bytes(): + module = _load_table_module() + for raw in ('{"x":"unfinished', '{', '[invalid'): + for fragments in ([raw], [raw[:2], raw[2:]]): + rows = [[{ + "kind": "tool_call", "name": "bad" if i == 0 else None, + "arguments": fragment, "complete": i == len(fragments) - 1, + }] for i, fragment in enumerate(fragments)] + rows.append([ + {"kind": "tool_call", "name": "echo", "arguments": '{"value":"é"}', "complete": True}, + {"kind": "tool_call", "name": "echo", "arguments": '{"value":"Café"}', "complete": True}, + ]) + expected = [ + {"kind": "tool_call", "name": "bad", "arguments": {}}, + {"kind": "tool_call", "name": "echo", "arguments": {"value": "é"}}, + {"kind": "tool_call", "name": "echo", "arguments": {"value": "Café"}}, + ] + assert module._assemble_stream(rows, recorded_assembly=expected) == expected + expected[0]["arguments"] = raw + assert module._assemble_stream(rows) == expected + + +def test_dynamo_preserves_valid_literal_string_fallback(): + module = _load_table_module() + rows = [[{"kind": "tool_call", "name": "bad", "arguments": '{"value":"unfinished"}', "complete": True}]] + assert module._assemble_stream(rows) == [ + {"kind": "tool_call", "name": "bad", "arguments": {"value": "unfinished"}}, + ] + + +def test_recorded_fallback_requires_matching_call_sequence(): + module = _load_table_module() + rows = [[ + {"kind": "tool_call", "name": "bad", "arguments": '{"x":"unfinished', "complete": True}, + {"kind": "text", "text": "between"}, + {"kind": "tool_call", "name": "echo", "arguments": '{"value":"é"}', "complete": True}, + ]] + raw = module._assemble_stream(rows) + assert module._assemble_stream(rows, recorded_assembly=raw) == raw + native = copy.deepcopy(raw) + native[0]["arguments"] = {} + assert module._assemble_stream(rows, recorded_assembly=native) == native + for mismatch in (native[:1], list(reversed(native)), [native[0], native[1], {**native[2], "arguments": {"value": "different"}}]): + assert module._assemble_stream(rows, recorded_assembly=mismatch) == raw + batch_order = [native[1], native[0], native[2]] + assert module._assemble_stream(rows, recorded_assembly=batch_order) == native diff --git a/conformance/utils/tests/test_unified_taxonomy_covers_corpus.py b/conformance/utils/tests/test_unified_taxonomy_covers_corpus.py index 8dcbe1d17..89896ba69 100644 --- a/conformance/utils/tests/test_unified_taxonomy_covers_corpus.py +++ b/conformance/utils/tests/test_unified_taxonomy_covers_corpus.py @@ -710,6 +710,7 @@ def test_every_authored_case_survives_emission_and_reload(): assert loaded["golden"] == case["golden"], f"{cid}: golden changed" assert loaded["init"] == case["init"], f"{cid}: init changed" assert loaded.get("tools") == case.get("tools"), f"{cid}: tools changed" + assert loaded.get("input_chunks") == case.get("input_chunks"), f"{cid}: chunk schedule changed" # --- counts live where they can be checked, not in registry prose --------------- @@ -725,24 +726,24 @@ def test_unified_case_counts_match_the_generator(): per_family = {fam: len(build_cases(fam)) for fam in FAMILIES} for fam in FAMILIES: family_specific = { - "deepseek_v4": 93, - "deepseek_v41": 93, - "gemma4": 95, - "glm47": 100, - "kimi_k2": 93, - "kimi_k3": 101, - "muse_glimmer": 94, - "qwen3": 93, + "deepseek_v4": 94, + "deepseek_v41": 94, + "gemma4": 96, + "glm47": 101, + "kimi_k2": 94, + "kimi_k3": 102, + "muse_glimmer": 95, + "qwen3": 94, }[fam] assert per_family[fam] == family_specific, f"{fam} diverged from the expected case count" - assert sum(per_family.values()) == 762 + assert sum(per_family.values()) == 770 def test_deferred_case_ids_are_not_in_the_active_taxonomy(): - deferred = {"1-2", "5-4", "5-5", "6-2", "30-14", "32-6", "50-1", "50-2"} | { + deferred = {"1-2", "5-5", "6-2", "30-14", "32-6", "50-1", "50-2"} | { f"31-{number}" for number in range(31, 41) } - assert len(UNIFIED_TAX) == 112 + assert len(UNIFIED_TAX) == 113 assert not {f"UNIFIED.{case_id}" for case_id in deferred} & { numbered_id(scenario) for scenario in UNIFIED_TAX } @@ -1125,7 +1126,52 @@ def test_kimi_k2_fixture_projection_rejects_an_empty_name(): assert _native_input_calls("kimi_k2", raw) == [] +def _assert_malformed_recovery(family, case): + raw = case["input"] + assert case["input_chunks"] == [raw] + assert case["init"] == {"starting_state": "None", "tool_output_mode": "Native", "named_tool": None} + openers = { + "kimi_k2": "<|tool_call_begin|>functions.echo:", + "kimi_k3": '<|open|>call tool="echo"', + "qwen3": "\n", + "glm47": "echo", + "deepseek_v4": '<｜DSML｜invoke name="echo">', + "deepseek_v41": '<｜DSML｜ invoke name="echo">', + "gemma4": "<|tool_call>call:echo{", + "muse_glimmer": "<|start|>assistant to=echo<|message|>", + } + at = raw.index(openers[family]) + prefix, suffix = raw[:at], raw[at:] + broken = 'x:<|"|>unfinished' if family == "gemma4" else '{"x":"unfinished' + assert broken in prefix and broken + '"' not in prefix + assert "bad" in prefix + closers = { + "kimi_k2": "<|tool_call_end|>", "kimi_k3": "<|close|>json<|sep|><|close|>call<|sep|>", + "qwen3": "\n\n", "glm47": "", + "deepseek_v4": "", + "deepseek_v41": "", + "gemma4": "}}", "muse_glimmer": "", + } + assert closers[family] in prefix + echoes = [{"kind": "tool_call", "name": "echo", "arguments": {"value": value}} for value in ("é", "Café")] + assert _native_input_calls(family, suffix) == echoes + bad = [] + if family == "kimi_k2": + bad = [{"kind": "tool_call", "name": "bad", "arguments": {}}] + elif family in {"qwen3", "glm47", "deepseek_v4", "muse_glimmer"}: + bad = [{"kind": "tool_call", "name": "bad", "arguments": {"value": '{"x":"unfinished'}}] + assert case["golden"] == bad + echoes + assert case["tools"] == [ + {"name": "bad", "parameters": {"type": "object", "properties": {"value": {"type": "object"}}}}, + {"name": "echo", "parameters": {"type": "object", "properties": {"value": {"type": "string"}}, "required": ["value"]}}, + ] + return echoes + + def _assert_input_carries_events(family: str, scenario: str, case: dict) -> None: + if scenario == "malformed_json_then_two_valid_calls": + _assert_malformed_recovery(family, case) + return raw = case["input"] tools = [event for event in case["golden"] if event["kind"] == "tool_call"] if tools: @@ -1229,7 +1275,8 @@ def _assert_cross_family_contract(corpus): assert init == {"starting_state": "Reasoning", "tool_output_mode": "Native", "named_tool": None} assert not case["input"].startswith(""), (family, scenario, "prefilled opener must be absent") init["starting_state"] = "None" - events = _logical_events(scenario, family, case["golden"]) + events = (_assert_malformed_recovery(family, case) if scenario == "malformed_json_then_two_valid_calls" + else _logical_events(scenario, family, case["golden"])) if scenario in {"arg_json_null", "arg_string_null"}: value = None if scenario == "arg_json_null" else "null" expected_type = "string" if value is not None else ["string", "null"] @@ -1354,3 +1401,22 @@ def test_dsml_eof_contract_rejects_invented_success_or_changed_body(family, muta case["input"] = case["input"].replace(old, new) with pytest.raises(AssertionError): _assert_cross_family_contract(corpus) + + +@pytest.mark.parametrize("family", FAMILIES) +@pytest.mark.parametrize("mutation", ["echo_order", "closed_string", "fallback", "chunks"]) +def test_malformed_recovery_contract_rejects_changed_stimulus(family, mutation): + case = build_cases(family)[f"UNIFIED.malformed_json_then_two_valid_calls.{family}"] + _assert_malformed_recovery(family, case) + if mutation == "echo_order": + case["input"] = case["input"].replace("Café", "OTHER").replace("é", "Café").replace("OTHER", "é") + case["input_chunks"] = [case["input"]] + elif mutation == "closed_string": + case["input"] = case["input"].replace("unfinished", 'unfinished"') + case["input_chunks"] = [case["input"]] + elif mutation == "fallback": + case["golden"][0]["arguments"] = {"value": "wrong"} + else: + case["input_chunks"] = [case["input"][:1], case["input"][1:]] + with pytest.raises(AssertionError): + _assert_malformed_recovery(family, case) diff --git a/conformance/utils/tests/test_unified_tool_schemas.py b/conformance/utils/tests/test_unified_tool_schemas.py index 82839c199..ec322c322 100644 --- a/conformance/utils/tests/test_unified_tool_schemas.py +++ b/conformance/utils/tests/test_unified_tool_schemas.py @@ -82,12 +82,17 @@ def test_schema_guard_corpus_keywords(schema, valid, invalid): def _assert_golden_schemas(cases, tools): - for case in cases.values(): + for case_id, case in cases.items(): offered_tools = case.get("tools", tools) schemas = {tool["name"]: tool["parameters"] for tool in offered_tools} assert len(schemas) == len(offered_tools) for event in case["golden"]: if event["kind"] == "tool_call": + if case_id.startswith("UNIFIED.malformed_json_then_two_valid_calls.") and event["name"] == "bad": + family = case_id.rsplit(".", 1)[1] + assert family in {"kimi_k2", "qwen3", "glm47", "deepseek_v4", "muse_glimmer"} + assert event["arguments"] == ({} if family == "kimi_k2" else {"value": '{"x":"unfinished'}) + continue _assert_value(event["arguments"], schemas[event["name"]]) @@ -147,7 +152,7 @@ def test_rust_harnesses_consume_the_shared_and_case_schema_owners(): assert "schemas.cloned().unwrap_or_else(unified_tool_schemas)" in common for name in ("unified_render.rs", "unified_parity.rs", "capture_cross_version.rs"): source = (tests / name).read_text() - assert "unified_tools as tools" in source + assert "unified_tools as tools" in source or "unified_tools_for_schemas" in source assert "fn tools()" not in source render = (tests / "unified_render.rs").read_text() parity = (tests / "unified_parity.rs").read_text() @@ -166,3 +171,10 @@ def test_recovery_successor_respects_string_schema(): assert case["golden"] == [{"kind": "tool_call", "name": "g", "arguments": {"y": "2"}}] assert G.k3_argument("y", "string", "2") in case["input"] assert G.k3_open("call", [("tool", "bad"), ("index", "1")]) + "not-an-argument" in case["input"] + + +def test_required_schema_rejects_missing_value(): + schema = {"type": "object", "properties": {"value": {"type": "string"}}, "required": ["value"]} + assert matches_schema({"value": "é"}, schema) + assert not matches_schema({}, schema) + assert not matches_schema({"value": None}, schema) diff --git a/parsers/v2-py/Cargo.toml b/parsers/v2-py/Cargo.toml index 33841c513..066af6e60 100644 --- a/parsers/v2-py/Cargo.toml +++ b/parsers/v2-py/Cargo.toml @@ -22,5 +22,5 @@ name = "dynamo_parsers_v2" crate-type = ["cdylib"] [dependencies] -dynamo-parsers-v2 = { path = "../v2", version = "0.7.14" } +dynamo-parsers-v2 = { path = "../v2", version = "0.7.15" } pyo3 = { workspace = true } diff --git a/parsers/v2/Cargo.toml b/parsers/v2/Cargo.toml index 7ab80f488..f0f231f56 100644 --- a/parsers/v2/Cargo.toml +++ b/parsers/v2/Cargo.toml @@ -2,7 +2,7 @@ # SPDX-License-Identifier: Apache-2.0 [package] name = "dynamo-parsers-v2" -version = "0.7.14" +version = "0.7.15" edition.workspace = true license = "Apache-2.0" description = "Token-incremental tool-call streaming parsers for Dynamo" diff --git a/parsers/v2/src/lib.rs b/parsers/v2/src/lib.rs index c2f9b37b9..a800a507d 100644 --- a/parsers/v2/src/lib.rs +++ b/parsers/v2/src/lib.rs @@ -35,7 +35,7 @@ pub use unified::{ InvalidGuidedPayloadPolicy, REGISTERED_UNIFIED_FAMILIES, UnifiedEvent, UnifiedParser, UnifiedParserEvent, UnifiedParserExt, UnifiedParserFactory, UnifiedParserInit, UnifiedParserOutput, UnifiedParserStartingState, UnifiedToolOutputMode, assemble, - builtin_unified_families, canonical_unified_family, create_unified_parser_for_family, - register_unified_parser, tool_arguments_raw, unregister_unified_parser, - vendor_unified_families, + assemble_with_tool_indices, builtin_unified_families, canonical_unified_family, + create_unified_parser_for_family, register_unified_parser, tool_arguments_raw, + unregister_unified_parser, vendor_unified_families, }; diff --git a/parsers/v2/src/tool_calling/scan.rs b/parsers/v2/src/tool_calling/scan.rs index 3205b0e80..c860ee78c 100644 --- a/parsers/v2/src/tool_calling/scan.rs +++ b/parsers/v2/src/tool_calling/scan.rs @@ -135,6 +135,34 @@ where .unwrap_or(0) } +/// Resumable JSON quote ownership for boundary scans and whitespace projection. +#[derive(Debug, Clone, Copy, Default)] +pub(crate) struct JsonStringState { + inside: bool, + escaped: bool, +} + +impl JsonStringState { + /// True when this character belongs to a quoted string, including its quotes. + pub(crate) fn advance(&mut self, character: char) -> bool { + if self.inside { + if self.escaped { + self.escaped = false; + } else if character == '\\' { + self.escaped = true; + } else if character == '"' { + self.inside = false; + } + true + } else if character == '"' { + self.inside = true; + true + } else { + false + } + } +} + /// Byte offset just past the first complete top-level JSON value (object or /// array) in `text`, skipping leading whitespace before it — or `None` if the /// value is absent, not object/array-shaped, or still open (unterminated @@ -163,21 +191,12 @@ pub(crate) fn json_value_end(text: &str) -> Option { // BEFORE a string that legitimately belongs to the same value, reopening // exactly that corruption. let mut stack = vec![if first == '{' { '}' } else { ']' }]; - let mut in_string = false; - let mut escape = false; + let mut quoted = JsonStringState::default(); for (idx, c) in chars { - if in_string { - if escape { - escape = false; - } else if c == '\\' { - escape = true; - } else if c == '"' { - in_string = false; - } + if quoted.advance(c) { continue; } match c { - '"' => in_string = true, '{' => stack.push('}'), '[' => stack.push(']'), '}' | ']' => { @@ -573,21 +592,9 @@ pub(crate) fn find_first_outside_strings<'a, I>(text: &str, markers: I) -> Optio where I: Clone + IntoIterator, { - let mut in_string = false; - let mut escape = false; + let mut quoted = JsonStringState::default(); for (idx, c) in text.char_indices() { - if in_string { - if escape { - escape = false; - } else if c == '\\' { - escape = true; - } else if c == '"' { - in_string = false; - } - continue; - } - if c == '"' { - in_string = true; + if quoted.advance(c) { continue; } if let Some((_, len)) = markers diff --git a/parsers/v2/src/unified/kimi_k2.rs b/parsers/v2/src/unified/kimi_k2.rs index 8f386b0ba..97fc013b5 100644 --- a/parsers/v2/src/unified/kimi_k2.rs +++ b/parsers/v2/src/unified/kimi_k2.rs @@ -558,6 +558,78 @@ mod tests { } } + #[test] + fn malformed_uuid_header_and_repeated_calls_keep_native_ids_across_reuse() { + let malformed = "<|tool_calls_section_begin|><|tool_call_begin|>call-5ec3039f-cc26-4d27-bd09-deedbe2b0c7b<|tool_call_argument_begin|>{\"city\":\"Paris\"}<|tool_call_end|><|tool_calls_section_end|>"; + for state in [ + UnifiedParserStartingState::None, + UnifiedParserStartingState::Reasoning, + ] { + for valid_count in [0, 1, 3] { + let reason = if state == UnifiedParserStartingState::Reasoning { + "check" + } else { + "check" + }; + let mut input = format!("{reason}before {malformed} after "); + for index in 0..valid_count { + input.push_str(&format!("<|tool_calls_section_begin|><|tool_call_begin|>functions.get_weather:{}<|tool_call_argument_begin|>{{\"city\":\"Paris\"}}<|tool_call_end|><|tool_calls_section_end|>", 7 + index * 2)); + } + let mut expected = vec![ + UnifiedEvent::Reasoning { + text: "check".into(), + }, + UnifiedEvent::Text { + text: "before after ".into(), + }, + ]; + for _ in 0..valid_count { + expected.push(UnifiedEvent::ToolCall { + name: "get_weather".into(), + arguments: serde_json::json!({"city":"Paris"}), + }); + } + for split in (0..=input.len()).filter(|&at| input.is_char_boundary(at)) { + let mut parser = kimi_k2_unified(&tools()); + for _ in 0..2 { + parser + .initialize_request(UnifiedParserInit { + starting_state: state, + ..Default::default() + }) + .unwrap(); + let mut events = parser.push(&input[..split]).unwrap(); + events.extend(parser.push(&input[split..]).unwrap()); + assert_eq!( + assemble(&events), + expected, + "state {state:?}, split {split}" + ); + events.extend(parser.finish().unwrap().events); + assert_eq!(assemble(&events), expected); + let calls: Vec<_> = events + .iter() + .filter_map(|event| match event { + UnifiedParserEvent::ToolCall(call) => Some(call), + _ => None, + }) + .collect(); + assert_eq!(calls.len(), valid_count); + for (index, call) in calls.iter().enumerate() { + assert_eq!(call.tool_index, index); + assert_eq!( + parser.tool_call_id(index), + Some(format!("functions.get_weather:{}", 7 + index * 2).as_str()) + ); + } + parser.reset(); + assert_eq!(parser.tool_call_id(0), None); + } + } + } + } + } + #[test] fn native_tool_call_id_survives_the_guided_router() { let input = "<|tool_calls_section_begin|><|tool_call_begin|>functions.get_weather:0<|tool_call_argument_begin|>{\"city\": \"Paris\"}<|tool_call_end|><|tool_calls_section_end|>"; diff --git a/parsers/v2/src/unified/kimi_k3.rs b/parsers/v2/src/unified/kimi_k3.rs index edf391478..a082fa32c 100644 --- a/parsers/v2/src/unified/kimi_k3.rs +++ b/parsers/v2/src/unified/kimi_k3.rs @@ -14,7 +14,7 @@ use serde_json::Value; use crate::tool_calling::scan::{ GuidedInvokePrefix, GuidedInvokePrefixContext, InvokeBoundary, InvokeBoundaryFactory, - find_first_outside_strings, json_value_end, marker_prefix_suffix_len, + JsonStringState, find_first_outside_strings, json_value_end, marker_prefix_suffix_len, }; use crate::tool_calling::traits::{Tool, ToolCallDelta}; use crate::unified::{ @@ -210,6 +210,7 @@ impl KimiK3HeaderScan { enum CallBoundary { Complete { body_end: usize, consumed: usize }, Recover { body_end: usize }, + SkipMalformed { consumed: usize }, Resync { at: usize }, Pending, Malformed, @@ -234,8 +235,20 @@ struct KimiK3CallBoundary { body_kind: CallBodyKind, root_call_open: Option, pending_call_open: Option, - complete_call_open: Option, - call_closes: Vec, + complete_call_opens: Vec, + pending_call_is_literal: bool, + literal_call_opens: Vec, + literal_before_outer_close: bool, + outer_call_closed: bool, + provisional_string_close: bool, + literal_argument_depths: Vec, + pending_argument_open: Option, + string_argument: bool, + json_argument: Option, + literal_json_argument: bool, + ignore_literal_json: bool, + argument_close_pending: bool, + call_closes: Vec, arg_opens: Vec, arg_closes: Vec, json_closes: Vec, @@ -260,7 +273,19 @@ impl KimiK3CallBoundary { body_kind: CallBodyKind::Unknown, root_call_open: None, pending_call_open: None, - complete_call_open: None, + complete_call_opens: Vec::new(), + pending_call_is_literal: false, + literal_call_opens: Vec::new(), + literal_before_outer_close: false, + outer_call_closed: false, + provisional_string_close: false, + literal_argument_depths: Vec::new(), + pending_argument_open: None, + string_argument: false, + json_argument: None, + literal_json_argument: false, + ignore_literal_json: false, + argument_close_pending: false, call_closes: Vec::new(), arg_opens: Vec::new(), arg_closes: Vec::new(), @@ -284,7 +309,24 @@ impl KimiK3CallBoundary { } fn advance(&mut self, text: &str, flush: bool) -> CallBoundary { - self.scan_appended(text, flush); + self.scan_appended(text, flush, false); + let reconsider_unterminated_json = + flush && self.json_argument.is_some() && !self.literal_json_argument; + if reconsider_unterminated_json { + // An unterminated JSON string can hide the structural close and the + // next call. At EOF, reuse this tracker with JSON string ownership + // disabled so only a structurally complete later call can resync. + self.reset_progress(); + self.scan_appended(text, true, true); + } + if flush && self.literal_json_argument && self.json_argument.is_some() { + // Invalid JSON in a quoted native passage is still outer string data. + // Only EOF proves its unmatched quote cannot later close; rescan the + // literal envelope without letting that quote swallow outer syntax. + self.reset_progress(); + self.ignore_literal_json = true; + self.scan_appended(text, true, false); + } let Some(header_len) = self.header_len else { return if flush { CallBoundary::Malformed @@ -293,9 +335,40 @@ impl KimiK3CallBoundary { }; }; - if let Some(next_call) = self.complete_call_open.filter(|at| *at > header_len) { - if let Some(call_close) = self - .call_closes + if reconsider_unterminated_json + && self.context == CallBoundaryContext::Guided + && let Some(consumed) = self.guided_malformed_envelope_end(text, header_len) + { + return CallBoundary::SkipMalformed { consumed }; + } + + // A channel fence settles balanced candidates without waiting for EOF. + // An unmatched embedded header may borrow the outer closers, so that + // remaining ambiguity still requires best-effort recovery at EOF. + let last_outer_close = self.call_closes.iter().rev().find(|close| !close.literal); + let recover_literal = !self.outer_call_closed + && (flush + || (!self.literal_before_outer_close || last_outer_close.is_some()) + && !self.outer_closes.is_empty() + && self.literal_argument_depths.is_empty()); + let call_opens = if recover_literal { + &self.literal_call_opens + } else { + &self.complete_call_opens + }; + let call_closes: Vec<_> = self + .call_closes + .iter() + .filter(|close| !close.literal || recover_literal) + .map(|close| close.hit) + .collect(); + let recovery_start = last_outer_close.map_or(header_len, |close| close.hit.end()); + for next_call in call_opens + .clone() + .into_iter() + .filter(|at| *at > header_len && (!recover_literal || *at >= recovery_start)) + { + if let Some(call_close) = call_closes .iter() .copied() .filter(|close| close.end() <= next_call) @@ -319,10 +392,10 @@ impl KimiK3CallBoundary { if let Some(body_end) = self.recovery_body_end(text, header_len, next_call) { return self.recover_at(text, header_len, body_end); } - if let Some(call_close) = self - .call_closes + if let Some(call_close) = call_closes .iter() .copied() + .rev() .find(|close| close.end() <= next_call) && let Some(arguments) = self.parse_body(&text[header_len..call_close.at]) { @@ -337,6 +410,10 @@ impl KimiK3CallBoundary { } } + if !recover_literal && !self.outer_call_closed && !self.literal_call_opens.is_empty() { + return CallBoundary::Pending; + } + if let Some((call_close, arguments)) = self.structural_call_close(text, header_len, flush) { self.completed_arguments = Some(arguments); return CallBoundary::Complete { @@ -380,7 +457,12 @@ impl KimiK3CallBoundary { header_len: usize, flush: bool, ) -> Option<(TokenHit, String)> { - let call_closes = self.call_closes.clone(); + let call_closes: Vec<_> = self + .call_closes + .iter() + .filter(|close| !close.literal || flush) + .map(|close| close.hit) + .collect(); for call_close in call_closes.into_iter().rev() { let boundary = self .outer_closes @@ -482,7 +564,11 @@ impl KimiK3CallBoundary { } fn recovery_body_end(&self, text: &str, header_len: usize, limit: usize) -> Option { - if !self.call_closes.is_empty() { + if self + .call_closes + .iter() + .any(|close| close.hit.end() <= limit) + { return None; } let candidate = match self.body_kind { @@ -545,25 +631,66 @@ impl KimiK3CallBoundary { }) } + fn guided_malformed_envelope_end(&mut self, text: &str, header_len: usize) -> Option { + if self.body_kind != CallBodyKind::RawJson { + return None; + } + let call_closes: Vec<_> = self + .call_closes + .iter() + .filter(|close| !close.literal) + .map(|close| close.hit) + .collect(); + let first_close = *call_closes.first()?; + if self.parse_body(&text[header_len..first_close.at]).is_some() { + return None; + } + for close in call_closes { + let Some(outer) = self + .outer_closes + .iter() + .copied() + .find(|outer| outer.at >= close.end()) + else { + continue; + }; + let suffix = text[outer.end()..].trim_start(); + let Some(end) = json_value_end(suffix) else { + continue; + }; + if serde_json::from_str::(&suffix[..end]).is_ok() + && suffix[end..].trim().is_empty() + { + return Some(outer.end()); + } + } + None + } + fn take_arguments(&mut self) -> Option { self.completed_arguments.take() } - fn scan_appended(&mut self, text: &str, flush: bool) { + fn scan_appended(&mut self, text: &str, flush: bool, ignore_json_strings: bool) { if text.len() < self.scanned { self.reset_progress(); } - let ends_at_boundary = SCANNER_MARKERS - .iter() - .any(|(_, marker)| marker.variants().any(|variant| text.ends_with(variant))); - let mut scan_limit = text - .len() - .saturating_sub((!flush && !ends_at_boundary) as usize * SCANNER_HOLDBACK); - while !text.is_char_boundary(scan_limit) { - scan_limit -= 1; - } - while self.scanned < scan_limit { - let suffix = &text[self.scanned..scan_limit]; + while self.scanned < text.len() { + let suffix = &text[self.scanned..]; + let character = suffix.chars().next().expect("non-empty scanner suffix"); + if self + .json_argument + .as_mut() + .filter(|_| !ignore_json_strings) + .is_some_and(|quoted| quoted.advance(character)) + { + self.scanned += character.len_utf8(); + #[cfg(test)] + { + self.scanned_bytes += character.len_utf8(); + } + continue; + } if !flush && scanner_marker_is_partial(suffix) { break; } @@ -572,7 +699,7 @@ impl KimiK3CallBoundary { at: self.scanned, len, }; - self.note_token(text, kind, hit); + self.note_token(text, kind, hit, ignore_json_strings); self.scanned += len; #[cfg(test)] { @@ -581,7 +708,6 @@ impl KimiK3CallBoundary { continue; } - let character = suffix.chars().next().expect("non-empty scanner suffix"); if self .header_len .is_some_and(|header_len| self.scanned >= header_len) @@ -590,6 +716,24 @@ impl KimiK3CallBoundary { { self.body_kind = CallBodyKind::Malformed; } + if !character.is_whitespace() + && self.pending_call_open.is_none() + && self.pending_argument_open.is_none() + && self.literal_argument_depths.is_empty() + { + self.argument_close_pending = false; + if self.provisional_string_close { + if matches!(self.return_channel, Mode::Response | Mode::Reasoning) { + // Keep the close provisional until a later argument close + // proves it was literal or a channel fence settles it. + self.outer_call_closed = false; + } else { + self.string_argument = true; + self.outer_call_closed = false; + self.provisional_string_close = false; + } + } + } self.scanned += character.len_utf8(); #[cfg(test)] { @@ -607,12 +751,44 @@ impl KimiK3CallBoundary { self.header_len = header_len; } - fn note_token(&mut self, text: &str, kind: ScannerToken, hit: TokenHit) { + fn note_token( + &mut self, + text: &str, + kind: ScannerToken, + hit: TokenHit, + ignore_json_strings: bool, + ) { + if kind == ScannerToken::ArgumentClose + && self.provisional_string_close + && matches!(self.return_channel, Mode::Response | Mode::Reasoning) + { + // A later argument close proves the earlier paired markers were + // inside the quoted value, so resume string ownership there. + self.provisional_string_close = false; + self.string_argument = true; + self.outer_call_closed = false; + } if kind == ScannerToken::CallOpen { + let provisional_response_close = self.provisional_string_close + && matches!(self.return_channel, Mode::Response | Mode::Reasoning); + if !provisional_response_close { + self.provisional_string_close = false; + } if self.root_call_open.is_none() { self.root_call_open = Some(hit.at); } else { + // A header inside an open argument can borrow the real outer + // closers. A channel fence alone cannot settle that ambiguity. + if self.string_argument + && !self.argument_close_pending + && !self.call_closes.iter().any(|close| !close.literal) + { + self.literal_before_outer_close = true; + } self.pending_call_open = Some(hit.at); + self.pending_call_is_literal = !self.literal_argument_depths.is_empty() + || self.string_argument + || provisional_response_close; } } if kind == ScannerToken::Sep { @@ -627,11 +803,79 @@ impl KimiK3CallBoundary { && let Some((_, len)) = parse_call_header(&text[at..]) && at + len == hit.end() { - self.complete_call_open = Some(at); + if self.pending_call_is_literal { + self.literal_argument_depths.push(0); + self.literal_call_opens.push(at); + } else { + self.complete_call_opens.push(at); + } self.pending_call_open = None; } } + // Native and Guided share the distinction between literal nested envelopes + // and an argument close followed by its owning call close. + let mut literal = !self.literal_argument_depths.is_empty(); + if let Some(depth) = self.literal_argument_depths.last_mut() { + match kind { + ScannerToken::ArgumentOpen => *depth += 1, + ScannerToken::ArgumentClose if *depth == 0 => { + self.literal_argument_depths.clear(); + literal = false; + } + ScannerToken::ArgumentClose => *depth -= 1, + ScannerToken::CallClose => { + self.literal_argument_depths.pop(); + } + _ => {} + } + } + if kind == ScannerToken::ArgumentOpen + && (literal || !self.string_argument || self.argument_close_pending) + { + self.pending_argument_open = Some(hit.at); + } + if kind == ScannerToken::Sep + && let Some(at) = self.pending_argument_open + && let Some((attrs, len)) = parse_tag_header(&text[at..], ARG_OPEN) + && at + len == hit.end() + { + let string_argument = attr_value(&attrs, "type").unwrap_or("string") == "string"; + self.json_argument = (!string_argument && (!literal || !self.ignore_literal_json)) + .then(JsonStringState::default); + self.literal_json_argument = literal && self.json_argument.is_some(); + if !literal { + self.string_argument = string_argument; + self.argument_close_pending = false; + } + self.pending_argument_open = None; + } + if kind == ScannerToken::JsonOpen + && (!self.string_argument || literal && !self.ignore_literal_json) + && !ignore_json_strings + { + self.json_argument = Some(JsonStringState::default()); + self.literal_json_argument = literal; + } else if matches!(kind, ScannerToken::ArgumentClose | ScannerToken::JsonClose) { + self.json_argument = None; + self.literal_json_argument = false; + } + if !literal { + if kind == ScannerToken::ArgumentClose { + self.argument_close_pending = true; + } + if kind == ScannerToken::CallClose + && (!self.string_argument || self.argument_close_pending) + { + // Following prose makes a paired argument/call close literal; + // only the next structural boundary commits this ownership. + self.provisional_string_close = self.string_argument; + self.string_argument = false; + self.argument_close_pending = false; + self.outer_call_closed = true; + } + } + let Some(header_len) = self.header_len else { return; }; @@ -647,8 +891,16 @@ impl KimiK3CallBoundary { if hit.at < header_len { return; } + if matches!( + kind, + ScannerToken::ToolsClose | ScannerToken::MessageClose | ScannerToken::EndOfMessage + ) || kind == ScannerToken::ThinkClose && self.return_channel == Mode::Reasoning + || kind == ScannerToken::ResponseClose && self.return_channel == Mode::Response + { + self.provisional_string_close = false; + } match kind { - ScannerToken::CallClose => self.call_closes.push(hit), + ScannerToken::CallClose => self.call_closes.push(CallCloseHit { hit, literal }), ScannerToken::ArgumentOpen => self.arg_opens.push(hit), ScannerToken::ArgumentClose => self.arg_closes.push(hit), ScannerToken::JsonClose => self.json_closes.push(hit), @@ -680,6 +932,12 @@ enum CallBodyKind { Malformed, } +#[derive(Debug, Clone, Copy)] +struct CallCloseHit { + hit: TokenHit, + literal: bool, +} + #[derive(Debug, Clone, Copy, PartialEq, Eq)] struct TokenHit { at: usize, @@ -723,8 +981,6 @@ const SCANNER_MARKERS: &[(ScannerToken, Marker)] = &[ (ScannerToken::Sep, SEP_MARKER), ]; -const SCANNER_HOLDBACK: usize = 32; - fn scanner_marker_at(text: &str) -> Option<(ScannerToken, usize)> { SCANNER_MARKERS .iter() @@ -814,6 +1070,7 @@ impl InvokeBoundary for KimiK3CallBoundary { match self.advance(candidate, flush) { CallBoundary::Complete { consumed, .. } => Some(consumed), CallBoundary::Recover { body_end } => Some(body_end), + CallBoundary::SkipMalformed { consumed } => Some(consumed), CallBoundary::Resync { .. } => None, CallBoundary::Pending | CallBoundary::Malformed => None, } @@ -845,6 +1102,7 @@ impl InvokeBoundary for KimiK3CallBoundary { CallBoundary::Resync { at } => Some(at), CallBoundary::Complete { .. } | CallBoundary::Recover { .. } + | CallBoundary::SkipMalformed { .. } | CallBoundary::Pending | CallBoundary::Malformed => None, } @@ -1081,6 +1339,19 @@ impl KimiK3Native { self.mode = Mode::Tools; true } + CallBoundary::SkipMalformed { consumed } => { + tracing::warn!( + why = "kimi_k3_malformed_call_body", + skipped_bytes = consumed, + "dropping malformed Kimi K3 call before guided JSON" + ); + self.buffer.drain(..consumed); + self.active_call = None; + self.call_header_scan = None; + self.call_boundary.reset(); + self.mode = Mode::Tools; + true + } CallBoundary::Pending if !flush => false, CallBoundary::Pending | CallBoundary::Malformed => { tracing::warn!( @@ -1408,7 +1679,10 @@ impl NativeUnified for KimiK3Native { match self.mode { Mode::Idle | Mode::Response => output.push_text(std::mem::take(&mut self.buffer)), Mode::Reasoning => output.push_reasoning(std::mem::take(&mut self.buffer)), - Mode::Tools | Mode::Call | Mode::Done => self.buffer.clear(), + Mode::Tools | Mode::Call | Mode::Done => { + self.buffer.clear(); + self.tools_open.clear(); + } } self.mode = Mode::Idle; self.active_call = None; @@ -1944,22 +2218,9 @@ fn encode_argument_value(arg_type: &str, raw: &str) -> String { fn compact_json(raw: &str) -> String { let mut output = String::with_capacity(raw.len()); - let mut in_string = false; - let mut escaped = false; + let mut quoted = JsonStringState::default(); for character in raw.chars() { - if in_string { - output.push(character); - if escaped { - escaped = false; - } else if character == '\\' { - escaped = true; - } else if character == '"' { - in_string = false; - } - } else if character == '"' { - in_string = true; - output.push(character); - } else if !character.is_whitespace() { + if quoted.advance(character) || !character.is_whitespace() { output.push(character); } } @@ -2858,6 +3119,267 @@ mod tests { ); } + #[test] + fn adjacent_calls_keep_each_boundary_at_every_split() { + for count in 1..=4 { + for repeated in [false, true] + .into_iter() + .filter(|&repeated| count > 1 || !repeated) + { + let mut body = String::new(); + let mut expected = Vec::new(); + for index in 1..=count { + let value = if repeated { + "Paris".into() + } else { + format!("city{index}") + }; + body.push_str(&call( + "get_weather", + &index.to_string(), + &arg("city", "string", &value), + )); + expected.push(UnifiedEvent::ToolCall { + name: "get_weather".into(), + arguments: serde_json::json!({"city": value}), + }); + } + let input = format!("{}{body}{}", TOOLS_OPEN.canonical, TOOLS_CLOSE.canonical); + assert_native_fragmentations(&input, &expected); + } + } + } + + #[test] + fn grouped_calls_progress_before_finish_and_survive_public_reuse() { + let body = (1..=3) + .map(|index| { + call( + "weather", + &index.to_string(), + &arg("city", "string", "Paris"), + ) + }) + .collect::(); + let envelope = format!("{}{body}{}", TOOLS_OPEN.canonical, TOOLS_CLOSE.canonical); + let expected = vec![ + UnifiedEvent::ToolCall { + name: "weather".into(), + arguments: serde_json::json!({"city":"Paris"}), + }; + 3 + ]; + for guided in [false, true] { + let input = if guided { + format!( + "{envelope}[{{\"name\":\"weather\",\"arguments\":{{\"city\":\"Paris\"}}}},{{\"name\":\"weather\",\"arguments\":{{\"city\":\"Paris\"}}}},{{\"name\":\"weather\",\"arguments\":{{\"city\":\"Paris\"}}}}]" + ) + } else { + envelope.clone() + }; + let mut parser = kimi_k3_unified(&[]); + for _ in 0..2 { + parser + .initialize_request(UnifiedParserInit { + tool_output_mode: if guided { + UnifiedToolOutputMode::GuidedJson { named_tool: None } + } else { + UnifiedToolOutputMode::Native + }, + ..UnifiedParserInit::default() + }) + .unwrap(); + let mut events = parser.push(&input).unwrap(); + assert_eq!( + assemble(&events), + expected, + "progress before finish, guided={guided}" + ); + if !guided { + for index in 0..3 { + assert_eq!( + parser.tool_call_id(index), + Some(format!("weather:{index}").as_str()) + ); + } + } + events.extend(parser.finish().unwrap().events); + assert_eq!(assemble(&events), expected); + assert_eq!(parser.reset(), ""); + assert_eq!(parser.tool_call_id(0), None); + } + } + } + + #[test] + fn typed_string_header_candidates_do_not_hide_later_calls() { + let literal = call("quoted", "8", &arg("nested", "string", "Zürich")); + let nested = call("quoted", "9", &arg("nested", "string", &literal)); + let raw = call( + "quoted", + "10", + "<|open|>json<|sep|>{\"x\":1}<|close|>json<|sep|>", + ); + for (value, spaced) in [ + (literal.clone(), false), + (literal.repeat(2), false), + (literal.repeat(2), true), + (nested.repeat(2), false), + (raw.repeat(2), false), + (format!("{}{literal}{literal}", ARG_CLOSE.canonical), false), + (format!("{}{literal}{literal}", CALL_CLOSE.canonical), false), + (format!("{}{literal}{literal}", TOOLS_CLOSE.canonical), false), + (format!("{}{}after{literal}{literal}", ARG_CLOSE.canonical, CALL_CLOSE.canonical), false), + (format!("{}{}after{literal}{literal}", ARG_CLOSE.canonical, CALL_CLOSE.canonical), true), + (format!("<|open|>argument key=\"nested\" type=\"number\"<|sep|>{literal}{literal}"), false), + ("<|open|>call tool=\"quoted\" index=\"8\"<|sep|><|open|>argument key=\"value\" type=\"string\"<|sep|>unfinished".into(), false), + ( + "<|open|>call tool=\"quoted\" index=\"8\"<|sep|>".into(), + false, + ), + ("<|open|>call tool=\"quoted\"".into(), false), + ] { + let value = format!("before{value}after"); + let value = if spaced { + value + .replace("<|open|>", "<|open|> ") + .replace("<|close|>", "<|close|> ") + .replace("<|sep|>", " <|sep|>") + } else { + value + }; + let arguments = serde_json::json!({"first":1, "value":value, "last":true}); + let body = format!( + "{}{}{}", + arg("first", "number", "1"), + arg("value", "string", &value), + arg("last", "boolean", "true") + ); + let first = call("echo", "1", &body); + let rest = format!( + "{}{}", + call("echo", "2", &arg("value", "string", "second")), + call("echo", "3", &arg("value", "string", "third")) + ); + let input = format!( + "{}{first}{rest}{}", + TOOLS_OPEN.canonical, TOOLS_CLOSE.canonical + ); + let expected = [ + UnifiedEvent::ToolCall { + name: "echo".into(), + arguments: arguments.clone(), + }, + UnifiedEvent::ToolCall { + name: "echo".into(), + arguments: serde_json::json!({"value":"second"}), + }, + UnifiedEvent::ToolCall { + name: "echo".into(), + arguments: serde_json::json!({"value":"third"}), + }, + ]; + assert_native_fragmentations(&input, &expected); + if value.contains("unfinished") { + assert_native_fragmentations(&format!("{first}{rest}"), &expected); + } + let guided = format!("{input}[{{\"name\":\"tail\",\"arguments\":{{}}}}]"); + assert_guided_all_utf8_fragmentations( + &guided, + UnifiedParserStartingState::None, + &[UnifiedEvent::ToolCall { + name: "tail".into(), + arguments: serde_json::json!({}), + }], + ); + for context in [CallBoundaryContext::Native, CallBoundaryContext::Guided] { + let mut boundary = KimiK3CallBoundary::new(context); + let (_, header_len) = parse_call_header(&first).unwrap(); + for _ in 0..2 { + boundary.begin(header_len, Mode::Tools); + let boundary_input = format!("{first}{rest}"); + let recover_at_eof = value.contains("unfinished"); + if recover_at_eof { + assert_eq!(boundary.advance(&boundary_input, false), CallBoundary::Pending); + assert_eq!( + boundary.advance(&format!("{boundary_input}{}", TOOLS_CLOSE.canonical), false), + CallBoundary::Pending + ); + } + assert_eq!( + boundary.advance(&boundary_input, recover_at_eof), + CallBoundary::Complete { + body_end: first.len() - CALL_CLOSE.canonical.len(), + consumed: first.len() + } + ); + assert_eq!( + serde_json::from_str::( + &boundary.take_arguments().unwrap() + ) + .unwrap(), + arguments + ); + } + } + } + } + + #[test] + fn embedded_call_ownership_drops_at_eof_and_resets_during_a_literal() { + let partial = format!( + "{}{}{}{}{}", + TOOLS_OPEN.canonical, + "<|open|>call tool=\"echo\" index=\"1\"<|sep|>", + "<|open|>argument key=\"value\" type=\"string\"<|sep|>", + "<|open|>call tool=\"quoted\" index=\"8\"<|sep|>", + "<|open|>argument key=\"value\" type=\"string\"<|sep|>unfinished" + ); + for guided in [false, true] { + for finish in [false, true] { + let mut parser = kimi_k3_unified(&[]); + let init = UnifiedParserInit { + tool_output_mode: if guided { + UnifiedToolOutputMode::GuidedJson { named_tool: None } + } else { + UnifiedToolOutputMode::Native + }, + ..UnifiedParserInit::default() + }; + parser.initialize_request(init.clone()).unwrap(); + assert!(parser.push(&partial).unwrap().is_empty()); + if finish { + if guided { + assert!(parser.finish().is_err()); + } else { + assert!(parser.finish().unwrap().events.is_empty()); + } + } + parser.reset(); + parser.initialize_request(init).unwrap(); + let input = if guided { + "[{\"name\":\"echo\",\"arguments\":{\"value\":\"ok\"}}]".into() + } else { + format!( + "{}{}{}", + TOOLS_OPEN.canonical, + call("echo", "1", &arg("value", "string", "ok")), + TOOLS_CLOSE.canonical + ) + }; + let mut events = parser.push(&input).unwrap(); + events.extend(parser.finish().unwrap().events); + assert_eq!( + assemble(&events), + vec![UnifiedEvent::ToolCall { + name: "echo".into(), + arguments: serde_json::json!({"value":"ok"}) + }] + ); + } + } + } + #[test] fn call_inside_reasoning_restores_reasoning_and_quarantines_reserved_structure() { let input = format!( diff --git a/parsers/v2/src/unified/mod.rs b/parsers/v2/src/unified/mod.rs index 11ffe3cd1..cba7cbb71 100644 --- a/parsers/v2/src/unified/mod.rs +++ b/parsers/v2/src/unified/mod.rs @@ -57,6 +57,8 @@ pub mod kimi_k2; pub mod kimi_k3; pub mod muse_glimmer; pub mod qwen3; +#[cfg(test)] +mod streaming_checkpoints; use std::collections::BTreeMap; use std::ops::Range; @@ -592,6 +594,17 @@ pub fn tool_arguments_raw(deltas: &[UnifiedParserEvent]) -> BTreeMap Vec { + assemble_with_tool_indices(deltas) + .into_iter() + .map(|(_, event)| event) + .collect() +} + +/// Assemble events while retaining each call's original parser index. +/// Text and reasoning carry no index; dropping an incomplete call never renumbers survivors. +pub fn assemble_with_tool_indices( + deltas: &[UnifiedParserEvent], +) -> Vec<(Option, UnifiedEvent)> { // Coalesce adjacent same-kind runs with the SAME helper the scan core uses, so // `I8` has exactly ONE implementation instead of one per type. let mut merged: Vec = Vec::new(); @@ -606,23 +619,28 @@ pub fn assemble(deltas: &[UnifiedParserEvent]) -> Vec { // Convert, joining each call's argument fragments. Keyed by `tool_index` so // fragments of two interleaved calls cannot merge, and carrying each call's // position so it stays where its FIRST delta landed. - let mut out: Vec = Vec::new(); + let mut out: Vec<(Option, UnifiedEvent)> = Vec::new(); let mut calls: BTreeMap = BTreeMap::new(); for delta in merged { match delta { - UnifiedParserEvent::Reasoning(text) => out.push(UnifiedEvent::Reasoning { text }), - UnifiedParserEvent::Text(text) => out.push(UnifiedEvent::Text { text }), + UnifiedParserEvent::Reasoning(text) => { + out.push((None, UnifiedEvent::Reasoning { text })) + } + UnifiedParserEvent::Text(text) => out.push((None, UnifiedEvent::Text { text })), UnifiedParserEvent::ToolCall(call) => { let (pos, raw, complete) = calls.entry(call.tool_index).or_insert_with(|| { - out.push(UnifiedEvent::ToolCall { - name: String::new(), - arguments: serde_json::Value::Null, - }); + out.push(( + Some(call.tool_index), + UnifiedEvent::ToolCall { + name: String::new(), + arguments: serde_json::Value::Null, + }, + )); (out.len() - 1, String::new(), false) }); raw.push_str(&call.arguments); if let Some(incoming) = call.name - && let UnifiedEvent::ToolCall { name, .. } = &mut out[*pos] + && let UnifiedEvent::ToolCall { name, .. } = &mut out[*pos].1 && name.is_empty() { *name = incoming; @@ -638,7 +656,7 @@ pub fn assemble(deltas: &[UnifiedParserEvent]) -> Vec { incomplete_positions.push(pos); continue; } - if let UnifiedEvent::ToolCall { arguments, .. } = &mut out[pos] { + if let UnifiedEvent::ToolCall { arguments, .. } = &mut out[pos].1 { // Best-effort (P3): a malformed payload must not take down the turn, but // it is NOT discarded silently — `{}` alone is indistinguishable from a // genuine no-arg call, so a corrupted argument would look like a clean parse. @@ -660,17 +678,17 @@ pub fn assemble(deltas: &[UnifiedParserEvent]) -> Vec { out.remove(pos); } let mut recoalesced = Vec::with_capacity(out.len()); - for event in out { + for (index, event) in out { match (recoalesced.last_mut(), event) { ( - Some(UnifiedEvent::Text { text: existing }), + Some((None, UnifiedEvent::Text { text: existing })), UnifiedEvent::Text { text: incoming }, ) => existing.push_str(&incoming), ( - Some(UnifiedEvent::Reasoning { text: existing }), + Some((None, UnifiedEvent::Reasoning { text: existing })), UnifiedEvent::Reasoning { text: incoming }, ) => existing.push_str(&incoming), - (_, event) => recoalesced.push(event), + (_, event) => recoalesced.push((index, event)), } } recoalesced @@ -5596,18 +5614,13 @@ mod tests { for chunk in chunks { parser.parse_into(chunk, &mut out).unwrap(); } - let before_finish = out.events.len(); - out.append(&mut parser.finish().unwrap()); assert_eq!( out.assembled(), want, - "{family} {state:?} {policy:?} named={named} schedule={schedule}" + "tool output must be available before EOF: {family} {state:?} {policy:?} named={named} schedule={schedule}" ); assert_eq!(out.events.iter().filter(|event| matches!(event, UnifiedParserEvent::ToolCall(delta) if delta.complete)).count(), 1); - assert!( - before_finish > 0, - "completed input must make pre-finish progress" - ); + assert!(parser.finish().unwrap().events.is_empty()); assert!(parser.finish().is_err()); assert!( parser @@ -6676,6 +6689,33 @@ mod tests { ); } + #[test] + fn indexed_assembly_keeps_sparse_interleaved_indices_after_drops() { + let mut incomplete = call(2, Some("drop"), "{"); + if let UnifiedParserEvent::ToolCall(ref mut call) = incomplete { + call.complete = false; + } + let deltas = vec![ + incomplete, + call(9, Some("f"), "{\"x\":"), + UnifiedParserEvent::Text("between".into()), + call(5, Some("f"), "{\"x\":1}"), + call(9, None, "1}"), + ]; + let indexed = assemble_with_tool_indices(&deltas); + assert_eq!( + indexed.iter().map(|(index, _)| *index).collect::>(), + vec![Some(9), None, Some(5)] + ); + assert_eq!( + indexed + .iter() + .map(|(_, event)| event.clone()) + .collect::>(), + assemble(&deltas) + ); + } + #[test] fn tool_only_projection_drops_order_but_not_bytes() { let result = ToolParseResult::from_deltas(vec![ diff --git a/parsers/v2/src/unified/streaming_checkpoints.rs b/parsers/v2/src/unified/streaming_checkpoints.rs new file mode 100644 index 000000000..2c3ff4812 --- /dev/null +++ b/parsers/v2/src/unified/streaming_checkpoints.rs @@ -0,0 +1,1103 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use super::*; + +struct Phase { + input: String, + events: Vec, + ids: Vec>, + eager_text: bool, +} + +fn text(value: &str) -> UnifiedEvent { + UnifiedEvent::Text { text: value.into() } +} + +fn reasoning(value: &str) -> UnifiedEvent { + UnifiedEvent::Reasoning { text: value.into() } +} + +fn call(value: &str) -> UnifiedEvent { + UnifiedEvent::ToolCall { + name: "echo".into(), + arguments: serde_json::json!({"value": value}), + } +} + +fn phase(input: impl Into, events: Vec, ids: &[&str]) -> Phase { + Phase { + input: input.into(), + events, + ids: ids.iter().map(|id| Some((*id).into())).collect(), + eager_text: false, + } +} + +fn safe_text(input: &str, events: Vec) -> Phase { + Phase { + eager_text: true, + ..phase(input, events, &[]) + } +} + +fn echo_tools() -> [Tool; 1] { + [Tool { + name: "echo".into(), + description: None, + strict: None, + parameters: serde_json::json!({"type":"object","properties":{"value":{"type":"string"}}}), + }] +} + +// The conformance table compares final assembly. These checkpoints instead reject +// output that arrives before ownership is settled or only after a later push/EOF. +fn assert_checkpoints( + family: &str, + init: UnifiedParserInit, + tools: &[Tool], + phases: &[Phase], + eof: &[UnifiedEvent], + eof_ids: &[&str], +) { + let mut schedules = vec![None, Some((usize::MAX, 0))]; + for (index, phase) in phases.iter().enumerate() { + schedules.extend( + (0..=phase.input.len()) + .filter(|at| phase.input.is_char_boundary(*at)) + .map(|at| Some((index, at))), + ); + } + for schedule in schedules { + let mut parser = create_unified_parser_for_family(family, tools).unwrap(); + for reuse in 0..2 { + parser.initialize_request(init.clone()).unwrap(); + let mut events = Vec::new(); + for (index, phase) in phases.iter().enumerate() { + let input = phase.input.as_str(); + let chunks = match schedule { + Some((usize::MAX, _)) => input + .char_indices() + .map(|(at, ch)| &input[at..at + ch.len_utf8()]) + .collect(), + Some((target, at)) if target == index => vec![&input[..at], &input[at..]], + _ => vec![input], + }; + let mut consumed = 0; + for chunk in chunks { + events.extend(parser.push(chunk).unwrap()); + consumed += chunk.len(); + let got = assemble(&events); + if phase.eager_text { + let mut expected = phase.events.clone(); + let last = expected.last_mut().unwrap(); + let (UnifiedEvent::Text { text } | UnifiedEvent::Reasoning { text }) = last + else { + panic!("safe-text phase needs text/reasoning") + }; + text.truncate(text.len() - input.len() + consumed); + if text.is_empty() { + expected.pop(); + } + assert_eq!( + got, expected, + "safe content delayed: {family} phase={index} {schedule:?}" + ); + } + assert!( + got.len() <= phase.events.len(), + "premature events: {family} phase={index} {schedule:?}: {got:?}" + ); + for (got, want) in got.iter().zip(&phase.events) { + match (got, want) { + ( + UnifiedEvent::Text { text: got }, + UnifiedEvent::Text { text: want }, + ) + | ( + UnifiedEvent::Reasoning { text: got }, + UnifiedEvent::Reasoning { text: want }, + ) => assert!( + want.starts_with(got), + "{family} phase={index}: {got:?} vs {want:?}" + ), + _ => assert_eq!( + got, want, + "premature/corrupt call: {family} phase={index} {schedule:?}" + ), + } + } + // Do not let assembly hide incomplete, duplicate, or ghost deltas. + let calls: Vec<_> = events + .iter() + .filter_map(|event| match event { + UnifiedParserEvent::ToolCall(delta) => Some(delta), + _ => None, + }) + .collect(); + assert!( + calls.len() <= phase.ids.len(), + "premature call delta: {family} phase={index} {schedule:?}" + ); + for (call_index, delta) in calls.iter().enumerate() { + assert!(delta.complete); + assert_eq!(delta.tool_index, call_index); + assert_eq!( + parser.tool_call_id(call_index), + phase.ids[call_index].as_deref() + ); + } + for absent in calls.len() + ..=phases + .iter() + .map(|phase| phase.ids.len()) + .max() + .unwrap_or(0) + + eof_ids.len() + { + assert_eq!( + parser.tool_call_id(absent), + None, + "ghost ID: {family} phase={index}" + ); + } + } + assert_eq!( + assemble(&events), + phase.events, + "delayed emission: {family} phase={index} {schedule:?} reuse={reuse}" + ); + for (index, id) in phase.ids.iter().enumerate() { + assert_eq!(parser.tool_call_id(index), id.as_deref()); + } + } + let finished = parser.finish().unwrap().events; + assert_eq!( + assemble(&finished), + eof, + "unexpected EOF output: {family} {schedule:?}" + ); + events.extend(finished); + let final_ids: Vec<_> = phases + .last() + .unwrap() + .ids + .iter() + .map(|id| id.as_deref()) + .chain(eof_ids.iter().copied().map(Some)) + .collect(); + let calls: Vec<_> = events + .iter() + .filter_map(|event| match event { + UnifiedParserEvent::ToolCall(delta) => Some(delta), + _ => None, + }) + .collect(); + assert_eq!( + calls.len(), + final_ids.len(), + "duplicate/unfinished EOF call" + ); + for (index, (delta, id)) in calls.iter().zip(&final_ids).enumerate() { + assert_eq!(delta.tool_index, index); + assert!(delta.complete); + assert_eq!(parser.tool_call_id(index), *id); + } + assert_eq!(parser.tool_call_id(final_ids.len()), None); + let mut expected = phases.last().unwrap().events.clone(); + expected.extend_from_slice(eof); + assert_eq!(assemble(&events), expected); + assert!(parser.finish().is_err()); + assert_eq!(parser.reset(), ""); + for index in 0..=final_ids.len() { + assert_eq!(parser.tool_call_id(index), None); + } + // Reset while syntax is pending as well as after a finished request. + parser.initialize_request(init.clone()).unwrap(); + assert!(parser.push("<").unwrap().is_empty()); + parser.reset(); + } + } +} + +#[test] +fn kimi_text_reasoning_checkpoints() { + for (family, open, close) in [ + ("kimi_k2", "", ""), + ("kimi_k3", "<|open|>think<|sep|>", "<|close|>think<|sep|>"), + ] { + let phases = [ + phase(format!("{open}ré"), vec![reasoning("ré")], &[]), + safe_text("界", vec![reasoning("ré界")]), + phase("<", vec![reasoning("ré界")], &[]), + phase("x", vec![reasoning("ré界", vec![reasoning("ré界 String { + format!( + "<|tool_call_begin|>functions.echo:{index}<|tool_call_argument_begin|>{{\"value\":\"é\"}}<|tool_call_end|>" + ) +} + +fn k3_call(index: usize, value: &str) -> String { + k3_named_call("echo", index, &k3_argument("value", "string", value)) +} + +#[test] +fn kimi_k2_recovery_and_repeated_call_checkpoints() { + let phases = [ + phase( + "<|tool_calls_section_begin|><|tool_call_begin|>call-5ec3039f-cc26-4d27-bd09-deedbe2b0c7b<|tool_call_argument_begin|>{\"value\":\"bad\"}<|tool_call_end|><|tool_calls_section_end|>", + vec![], + &[], + ), + phase(k2_call(7), vec![call("é")], &["functions.echo:7"]), + phase( + k2_call(9), + vec![call("é"); 2], + &["functions.echo:7", "functions.echo:9"], + ), + phase( + k2_call(11), + vec![call("é"); 3], + &["functions.echo:7", "functions.echo:9", "functions.echo:11"], + ), + ]; + assert_checkpoints( + "kimi_k2", + UnifiedParserInit::default(), + &echo_tools(), + &phases, + &[], + &[], + ); +} + +#[test] +fn kimi_k3_repeated_call_checkpoints() { + let second = k3_call(9, "é"); + let third = k3_call(11, "é"); + let header_end = second.find("<|sep|>").unwrap() + "<|sep|>".len(); + let third_header_end = third.find("<|sep|>").unwrap() + "<|sep|>".len(); + let phases = [ + phase( + format!("<|open|>tools<|sep|>{}", k3_call(7, "é")), + vec![], + &[], + ), + phase(&second[..header_end], vec![call("é")], &["echo:6"]), + phase(&second[header_end..], vec![call("é")], &["echo:6"]), + phase( + &third[..third_header_end], + vec![call("é"); 2], + &["echo:6", "echo:8"], + ), + phase( + &third[third_header_end..], + vec![call("é"); 2], + &["echo:6", "echo:8"], + ), + phase( + "<|close|>tools<|sep|>", + vec![call("é"); 3], + &["echo:6", "echo:8", "echo:10"], + ), + ]; + assert_checkpoints( + "kimi_k3", + UnifiedParserInit::default(), + &echo_tools(), + &phases, + &[], + &[], + ); +} + +#[test] +fn kimi_k3_embedded_call_checkpoints() { + // Typed strings have no escape for argument/call closers. Even a complete + // call-shaped passage can still be data; emitting it cannot be retracted. + // Following outer structure, not the apparent closer alone, settles ownership. + let literal = k3_call(8, "quoted").repeat(2); + let start = "<|open|>tools<|sep|><|open|>call tool=\"echo\" index=\"1\"<|sep|><|open|>argument key=\"value\" type=\"string\"<|sep|>before"; + let value = format!("before{literal}after"); + let phases = [ + phase(start, vec![], &[]), + phase(literal, vec![], &[]), + phase( + "after<|close|>argument<|sep|><|close|>call<|sep|>", + vec![], + &[], + ), + phase(k3_call(2, "é"), vec![call(&value)], &["echo:0"]), + phase( + "<|close|>tools<|sep|>", + vec![call(&value), call("é")], + &["echo:0", "echo:1"], + ), + ]; + assert_checkpoints( + "kimi_k3", + UnifiedParserInit::default(), + &echo_tools(), + &phases, + &[], + &[], + ); +} + +#[test] +fn kimi_guided_tool_checkpoints() { + for family in ["kimi_k2", "kimi_k3"] { + for named in [false, true] { + let init = UnifiedParserInit { + tool_output_mode: UnifiedToolOutputMode::GuidedJson { + named_tool: named.then(|| "echo".into()), + }, + ..Default::default() + }; + let payload = if named { + "{\"value\":\"é\"}" + } else { + "[{\"name\":\"echo\",\"arguments\":{\"value\":\"é\"}}]" + }; + let mut checkpoint = phase(payload, vec![call("é")], &[]); + checkpoint.ids.push(None); + assert_checkpoints(family, init, &echo_tools(), &[checkpoint], &[], &[]); + } + } +} + +#[test] +fn kimi_k3_malformed_then_valid_checkpoints() { + let phases = [ + phase( + "<|open|>tools<|sep|><|open|>call tool=\"bad\" index=\"1\"<|sep|>not-an-argument", + vec![], + &[], + ), + phase(k3_call(2, "é"), vec![], &[]), + phase("<|close|>tools<|sep|>", vec![call("é")], &["echo:1"]), + ]; + assert_checkpoints( + "kimi_k3", + UnifiedParserInit::default(), + &echo_tools(), + &phases, + &[], + &[], + ); +} + +#[test] +fn kimi_k3_malformed_json_calls_resync_only_at_eof() { + let malformed_object = r#"{"value":"unfinished\"#; + let malformed_array = r#"["unfinished é"#; + let raw_json = + format!("<|open|>json type=\"object\"<|sep|>{malformed_object}<|close|>json<|sep|>"); + let mut malformed_bodies = vec![(raw_json, None)]; + for kind in ["object", "array"] { + let malformed_value = if kind == "object" { + malformed_object + } else { + malformed_array + }; + for position in 0..3 { + let mut fields = Vec::new(); + let mut arguments = serde_json::Map::new(); + for index in 0..position { + let key = format!("before{index}"); + let value = format!("pré{index}"); + fields.push(k3_argument(&key, "string", &value)); + arguments.insert(key, serde_json::Value::String(value)); + } + fields.push(k3_argument("payload", kind, malformed_value)); + arguments.insert( + "payload".into(), + serde_json::Value::String(malformed_value.into()), + ); + for index in position..2 { + let key = format!("after{index}"); + let value = format!("Café{index}"); + fields.push(k3_argument(&key, "string", &value)); + arguments.insert(key, serde_json::Value::String(value)); + } + malformed_bodies.push((fields.concat(), Some(serde_json::Value::Object(arguments)))); + } + } + for (body, fallback) in malformed_bodies { + for count in 1..=2 { + let malformed = k3_named_call("bad", 1, &body); + let valid: Vec<_> = ["é", "Café"] + .into_iter() + .take(count) + .enumerate() + .map(|(offset, value)| { + k3_named_call("echo", offset + 2, &k3_argument("value", "string", value)) + }) + .collect(); + let phases = [ + phase(format!("<|open|>tools<|sep|>{malformed}"), vec![], &[]), + phase( + format!("{}<|close|>tools<|sep|>", valid.concat()), + vec![], + &[], + ), + ]; + let mut eof = Vec::new(); + let mut eof_ids = Vec::new(); + if let Some(fallback) = &fallback { + eof.push(UnifiedEvent::ToolCall { + name: "bad".into(), + arguments: fallback.clone(), + }); + eof_ids.push("bad:0"); + } + for value in ["é", "Café"].into_iter().take(count) { + eof.push(call(value)); + eof_ids.push(if value == "é" { "echo:1" } else { "echo:2" }); + } + assert_checkpoints( + "kimi_k3", + UnifiedParserInit::default(), + &echo_tools(), + &phases, + &eof, + &eof_ids, + ); + } + } +} + +#[test] +fn kimi_k3_unfinished_json_string_markers_alone_never_dispatch() { + for suffix in [ + "<|close|>call<|sep|>", + "<|open|>call tool=\"ghost\" index=\"8\"<|sep|>incomplete", + ] { + let malformed = format!( + "<|open|>tools<|sep|><|open|>call tool=\"bad\" index=\"1\"<|sep|><|open|>json type=\"object\"<|sep|>{{\"value\":\"unfinished{suffix}" + ); + assert_checkpoints( + "kimi_k3", + UnifiedParserInit::default(), + &echo_tools(), + &[phase(malformed, vec![], &[])], + &[], + &[], + ); + } +} + +#[test] +fn kimi_k3_unfinished_json_string_keeps_complete_literal_calls_inert() { + let malformed = format!( + "<|open|>tools<|sep|>{}<|open|>json type=\"object\"<|sep|>{{\"value\":\"unfinished{}{}<|close|>json<|sep|><|close|>call<|sep|>", + "<|open|>call tool=\"bad\" index=\"1\"<|sep|>", + k3_call(8, "literal"), + k3_call(9, "literal2"), + ); + let valid = k3_call(2, "real"); + let phases = [ + phase(malformed, vec![], &[]), + phase(format!("{valid}<|close|>tools<|sep|>"), vec![], &[]), + ]; + assert_checkpoints( + "kimi_k3", + UnifiedParserInit::default(), + &echo_tools(), + &phases, + &[call("real")], + &["echo:1"], + ); +} + +#[test] +fn kimi_k3_malformed_json_recovers_calls_after_the_wrapper_boundary() { + let malformed = "<|open|>tools<|sep|><|open|>call tool=\"bad\" index=\"1\"<|sep|><|open|>json type=\"object\"<|sep|>{\"value\":\"unfinished<|close|>json<|sep|><|close|>call<|sep|>"; + let later_calls = [ + k3_call(8, "literal"), + k3_call(9, "literal2"), + k3_call(2, "real"), + ] + .concat(); + let phases = [ + phase(malformed, vec![], &[]), + phase(format!("{later_calls}<|close|>tools<|sep|>"), vec![], &[]), + ]; + assert_checkpoints( + "kimi_k3", + UnifiedParserInit::default(), + &echo_tools(), + &phases, + &[call("literal"), call("literal2"), call("real")], + &["echo:7", "echo:8", "echo:1"], + ); +} + +#[test] +fn kimi_k3_unmatched_embedded_header_waits_for_eof() { + // An unmatched quoted header can borrow the outer closers. Only EOF permits + // best-effort recovery; the ordinary balanced case above must progress sooner. + let value = "before<|open|>call tool=\"quoted\" index=\"8\"<|sep|><|open|>argument key=\"value\" type=\"string\"<|sep|>unfinished"; + let phases = [ + phase( + format!("<|open|>tools<|sep|>{}", k3_call(1, value)), + vec![], + &[], + ), + phase( + format!("{}<|close|>tools<|sep|>", k3_call(2, "é")), + vec![], + &[], + ), + ]; + assert_checkpoints( + "kimi_k3", + UnifiedParserInit::default(), + &echo_tools(), + &phases, + &[call(value), call("é")], + &["echo:0", "echo:1"], + ); +} + +#[test] +fn kimi_k3_open_response_checkpoints() { + for prefilled in [false, true] { + let init = UnifiedParserInit { + starting_state: if prefilled { + UnifiedParserStartingState::Response + } else { + UnifiedParserStartingState::None + }, + ..Default::default() + }; + let phases = [ + phase( + if prefilled { + "" + } else { + "<|open|>response<|sep|>" + }, + vec![], + &[], + ), + safe_text("visible é", vec![text("visible é")]), + safe_text("界", vec![text("visible é界")]), + phase("<", vec![text("visible é界")], &[]), + phase("x", vec![text("visible é界response<|sep|", vec![text("visible é界", vec![text("visible é界 String { + format!("<|open|>argument key=\"{key}\" type=\"{kind}\"<|sep|>{value}<|close|>argument<|sep|>") +} + +fn k3_named_call(name: &str, index: usize, body: &str) -> String { + format!("<|open|>call tool=\"{name}\" index=\"{index}\"<|sep|>{body}<|close|>call<|sep|>") +} + +fn weather_call() -> UnifiedEvent { + UnifiedEvent::ToolCall { + name: "get_weather".into(), + arguments: serde_json::json!({"city":"Paris"}), + } +} + +fn review_tools() -> Vec { + [ + ("search", serde_json::json!({"q":{"type":"string"}})), + ("get_weather", serde_json::json!({"city":{"type":"string"}})), + ("write_file", serde_json::json!({"path":{"type":"string"},"data":{"type":"object"},"content":{"type":"string"}})), + ].into_iter().map(|(name, properties)| Tool { + name: name.into(), description: None, strict: None, + parameters: serde_json::json!({"type":"object","properties":properties}), + }).collect() +} + +#[test] +fn kimi_k3_prose_between_calls_resolves_at_channel_fence() { + let first = k3_named_call("search", 1, &k3_argument("q", "string", "Paris")); + let second = k3_named_call("get_weather", 2, &k3_argument("city", "string", "Paris")); + let header_end = second.find("<|sep|>").unwrap() + "<|sep|>".len(); + let first_event = UnifiedEvent::ToolCall { + name: "search".into(), + arguments: serde_json::json!({"q":"Paris"}), + }; + for (channel, starting_state) in [ + ("tools", UnifiedParserStartingState::None), + ("response", UnifiedParserStartingState::None), + ("response", UnifiedParserStartingState::Response), + ("think", UnifiedParserStartingState::None), + ("think", UnifiedParserStartingState::Reasoning), + ] { + let mut expected = vec![first_event.clone()]; + if channel != "tools" { + expected.push(if channel == "think" { + reasoning("between é") + } else { + text("between é") + }); + } + expected.push(weather_call()); + let start = if starting_state == UnifiedParserStartingState::None { + format!("<|open|>{channel}<|sep|>{first}") + } else { + first.clone() + }; + let phases = [ + phase(start, vec![], &[]), + phase("between é", vec![], &[]), + phase(&second[..header_end], vec![], &[]), + phase(&second[header_end..], vec![], &[]), + phase( + format!("<|close|>{channel}<|sep|>"), + expected.clone(), + &["search:0", "get_weather:1"], + ), + ]; + assert_checkpoints( + "kimi_k3", + UnifiedParserInit { + starting_state, + tool_output_mode: UnifiedToolOutputMode::Native, + ..Default::default() + }, + &review_tools(), + &phases, + &[], + &[], + ); + } +} + +#[test] +fn kimi_k3_call_close_survives_response_or_reasoning_prose() { + for (starting_state, channel_close, prose) in [ + ( + UnifiedParserStartingState::Response, + "<|close|>response<|sep|>", + text("ordinary response prose"), + ), + ( + UnifiedParserStartingState::Reasoning, + "<|close|>think<|sep|>", + reasoning("ordinary reasoning prose"), + ), + ] { + let phases = [ + phase( + k3_named_call("echo", 1, &k3_argument("value", "string", "é")), + vec![], + &[], + ), + phase( + match prose.clone() { + UnifiedEvent::Text { text } | UnifiedEvent::Reasoning { text } => text, + _ => unreachable!(), + }, + vec![], + &[], + ), + phase(channel_close, vec![call("é"), prose], &["echo:0"]), + ]; + assert_checkpoints( + "kimi_k3", + UnifiedParserInit { + starting_state, + tool_output_mode: UnifiedToolOutputMode::Native, + ..Default::default() + }, + &echo_tools(), + &phases, + &[], + &[], + ); + } +} + +#[test] +fn kimi_k3_response_string_keeps_complete_literal_call_passages() { + let literal = k3_named_call("quoted", 8, &k3_argument("value", "string", "literal")); + let value = format!("before{literal}after"); + let phases = [ + phase( + k3_named_call("echo", 1, &k3_argument("value", "string", &value)), + vec![], + &[], + ), + phase("<|close|>response<|sep|>", vec![call(&value)], &["echo:0"]), + ]; + assert_checkpoints( + "kimi_k3", + UnifiedParserInit { + starting_state: UnifiedParserStartingState::Response, + tool_output_mode: UnifiedToolOutputMode::Native, + ..Default::default() + }, + &echo_tools(), + &phases, + &[], + &[], + ); +} + +#[test] +fn kimi_k3_json_string_keeps_complete_literal_call_passages() { + let literal = k3_named_call("quoted", 8, &k3_argument("value", "string", "literal")); + let value = format!("before{literal}after"); + let json = serde_json::json!({"value": value}).to_string(); + let body = format!("<|open|>json type=\"object\"<|sep|>{json}<|close|>json<|sep|>"); + let phases = [ + phase( + format!("<|open|>tools<|sep|>{}", k3_named_call("echo", 1, &body)), + vec![], + &[], + ), + phase("<|close|>tools<|sep|>", vec![call(&value)], &["echo:0"]), + ]; + assert_checkpoints( + "kimi_k3", + UnifiedParserInit::default(), + &echo_tools(), + &phases, + &[], + &[], + ); +} + +#[test] +fn kimi_k3_json_markers_and_nested_calls_keep_payload_ownership() { + let marker = "<|close|>call<|sep|><|open|>call<|sep|>"; + let pair = "<|close|>argument<|sep|><|close|>call<|sep|>"; + let second = k3_named_call("get_weather", 2, &k3_argument("city", "string", "Paris")); + for data in [ + serde_json::json!({"text":marker}), + serde_json::json!({"text":format!("before{pair}after")}), + serde_json::json!({"text":format!("quote \\\" é {marker}")}), + ] { + let raw = serde_json::to_string(&data).unwrap(); + for body in [ + k3_argument("data", "object", &raw), + format!( + "{}{}", + k3_argument("path", "string", "a.json"), + k3_argument("data", "object", &raw) + ), + ] { + let first = k3_named_call("write_file", 1, &body); + let arguments = if body.contains("key=\"path\"") { + serde_json::json!({"path":"a.json","data":data}) + } else { + serde_json::json!({"data":data}) + }; + let expected = vec![ + UnifiedEvent::ToolCall { + name: "write_file".into(), + arguments, + }, + weather_call(), + ]; + let input = format!("<|open|>tools<|sep|>{first}{second}<|close|>tools<|sep|>"); + assert_checkpoints( + "kimi_k3", + UnifiedParserInit { + tool_output_mode: UnifiedToolOutputMode::Native, + ..Default::default() + }, + &review_tools(), + &[phase( + &input, + expected.clone(), + &["write_file:0", "get_weather:1"], + )], + &[], + &[], + ); + } + let nested = k3_named_call("write_file", 8, &k3_argument("data", "object", &raw)); + for count in 1..=2 { + let value = format!("before{}after", nested.repeat(count)); + let input = format!( + "<|open|>tools<|sep|>{}{}<|close|>tools<|sep|>", + k3_call(1, &value), + k3_call(2, "é") + ); + assert_checkpoints( + "kimi_k3", + UnifiedParserInit { + tool_output_mode: UnifiedToolOutputMode::Native, + ..Default::default() + }, + &echo_tools(), + &[phase( + &input, + vec![call(&value), call("é")], + &["echo:0", "echo:1"], + )], + &[], + &[], + ); + } + } +} + +#[test] +fn kimi_k3_json_open_in_string_does_not_own_closing_markers() { + for suffix in ["\"unfinished", "{\"x\":\"unfinished", "[\"unfinished"] { + let value = format!("before<|open|>json<|sep|>{suffix}"); + let phases = [ + phase( + format!("<|open|>tools<|sep|>{}", k3_call(1, &value)), + vec![], + &[], + ), + phase( + format!("{}<|close|>tools<|sep|>", k3_call(2, "é")), + vec![call(&value), call("é")], + &["echo:0", "echo:1"], + ), + ]; + assert_checkpoints( + "kimi_k3", + UnifiedParserInit { + tool_output_mode: UnifiedToolOutputMode::Native, + ..Default::default() + }, + &echo_tools(), + &phases, + &[], + &[], + ); + } +} + +#[test] +fn kimi_k3_missing_call_close_recovers_before_eof_without_borrowing_next_call() { + let first = format!( + "<|open|>call tool=\"search\" index=\"1\"<|sep|>{}", + k3_argument("q", "string", "Paris") + ); + let second = k3_named_call("get_weather", 2, &k3_argument("city", "string", "Paris")); + let phases = [ + phase(format!("<|open|>tools<|sep|>{first}"), vec![], &[]), + phase(second, vec![], &[]), + phase( + "<|close|>tools<|sep|>", + vec![ + UnifiedEvent::ToolCall { + name: "search".into(), + arguments: serde_json::json!({"q":"Paris"}), + }, + weather_call(), + ], + &["search:0", "get_weather:1"], + ), + ]; + assert_checkpoints( + "kimi_k3", + UnifiedParserInit { + tool_output_mode: UnifiedToolOutputMode::Native, + ..Default::default() + }, + &review_tools(), + &phases, + &[], + &[], + ); +} + +#[test] +fn kimi_k3_balanced_literal_calls_before_narration_remain_argument_data() { + let literal = k3_named_call("quoted", 8, &k3_argument("value", "string", "Zurich")); + for count in 1..=2 { + for prefix in ["", "<|close|>argument<|sep|><|close|>call<|sep|>prose"] { + let value = format!("{prefix}{}", literal.repeat(count)); + let phases = [ + phase( + format!("<|open|>tools<|sep|>{}", k3_call(1, &value)), + vec![], + &[], + ), + phase(" narration ", vec![], &[]), + phase( + format!("{}<|close|>tools<|sep|>", k3_call(2, "é")), + vec![call(&value), call("é")], + &["echo:0", "echo:1"], + ), + ]; + assert_checkpoints( + "kimi_k3", + UnifiedParserInit::default(), + &echo_tools(), + &phases, + &[], + &[], + ); + } + } +} + +#[test] +fn kimi_k3_guided_native_wrappers_release_json_before_eof() { + let object = serde_json::json!({"text":"<|close|>call<|sep|><|open|>call<|sep|>"}).to_string(); + let wrappers = [ + format!("{} narration {}", k3_call(1, "Paris"), k3_call(2, "é")), + k3_named_call("write_file", 1, &k3_argument("data", "object", &object)), + k3_call(1, "before<|open|>json<|sep|>\"unfinished"), + ]; + for (index, wrapper) in wrappers.into_iter().enumerate() { + for named in [false, true] { + let payload = if named { + "{\"value\":\"é\"}" + } else { + "[{\"name\":\"echo\",\"arguments\":{\"value\":\"é\"}}]" + }; + let mut output = phase(payload, vec![call("é")], &[]); + output.ids.push(None); + let prefix = if index == 0 { + vec![text(" narration ")] + } else { + vec![] + }; + output.events.splice(0..0, prefix.clone()); + let phases = [ + phase( + format!("<|open|>tools<|sep|>{wrapper}<|close|>tools<|sep|>"), + prefix, + &[], + ), + output, + ]; + assert_checkpoints( + "kimi_k3", + UnifiedParserInit { + tool_output_mode: UnifiedToolOutputMode::GuidedJson { + named_tool: named.then(|| "echo".into()), + }, + ..Default::default() + }, + &echo_tools(), + &phases, + &[], + &[], + ); + } + } +} + +#[test] +fn kimi_k3_guided_malformed_raw_json_recovers_later_payload_at_eof() { + let malformed = k3_named_call( + "bad", + 1, + "<|open|>json type=\"object\"<|sep|>{\"value\":\"unfinished<|close|>json<|sep|>", + ); + for named in [false, true] { + let (payload, expected) = if named { + ("{\"value\":\"é\"}".to_string(), vec![call("é")]) + } else { + ( + "[{\"name\":\"echo\",\"arguments\":{\"value\":\"é\"}},{\"name\":\"echo\",\"arguments\":{\"value\":\"Café\"}}]".to_string(), + vec![call("é"), call("Café")], + ) + }; + let input = format!("<|open|>tools<|sep|>{malformed}<|close|>tools<|sep|>{payload}"); + let splits: Vec<_> = input + .char_indices() + .map(|(at, _)| at) + .chain([input.len()]) + .collect(); + for split in splits { + let mut parser = create_unified_parser_for_family("kimi_k3", &[]).unwrap(); + parser + .initialize_request(UnifiedParserInit { + tool_output_mode: UnifiedToolOutputMode::GuidedJson { + named_tool: named.then(|| "echo".into()), + }, + invalid_guided_payload: InvalidGuidedPayloadPolicy::RecoverAsText, + ..Default::default() + }) + .unwrap(); + let mut events = parser.push(&input[..split]).unwrap(); + assert!(events.is_empty(), "premature guided dispatch before EOF"); + events.extend(parser.push(&input[split..]).unwrap()); + assert!(events.is_empty(), "premature guided dispatch before EOF"); + assert_eq!(assemble(&parser.finish().unwrap().events), expected); + assert_eq!(parser.tool_call_id(0), None); + assert_eq!(parser.tool_call_id(1), None); + assert!(parser.finish().is_err()); + assert!(parser.reset().is_empty()); + } + + let mut parser = create_unified_parser_for_family("kimi_k3", &[]).unwrap(); + parser + .initialize_request(UnifiedParserInit { + tool_output_mode: UnifiedToolOutputMode::GuidedJson { + named_tool: named.then(|| "echo".into()), + }, + invalid_guided_payload: InvalidGuidedPayloadPolicy::RecoverAsText, + ..Default::default() + }) + .unwrap(); + for (at, character) in input.char_indices() { + assert!( + parser + .push(&input[at..at + character.len_utf8()]) + .unwrap() + .is_empty() + ); + } + assert_eq!(assemble(&parser.finish().unwrap().events), expected); + assert_eq!(parser.tool_call_id(0), None); + assert_eq!(parser.tool_call_id(1), None); + assert!(parser.finish().is_err()); + assert!(parser.reset().is_empty()); + } +} + +#[test] +fn kimi_k3_malformed_json_in_literal_call_recovers_at_eof() { + // A quoted native passage may contain invalid JSON. Its unmatched quote + // cannot seize the outer argument; EOF permits recovery without dispatching it. + for body in [ + "<|open|>json<|sep|>{\"x\":\"unfinished<|close|>json<|sep|>".to_string(), + k3_argument("data", "object", "{\"x\":\"unfinished"), + k3_argument("data", "array", "[\"unfinished"), + ] { + let value = k3_named_call("quoted", 8, &body); + let input = format!( + "<|open|>tools<|sep|>{}{}<|close|>tools<|sep|>", + k3_call(1, &value), + k3_call(2, "é") + ); + assert_checkpoints( + "kimi_k3", + UnifiedParserInit::default(), + &echo_tools(), + &[phase(input, vec![], &[])], + &[call(&value), call("é")], + &["echo:0", "echo:1"], + ); + } +}