[xianlubird]

Overview

Messages streaming waits for the turn-level tool finish before publishing any tool blocks, even when the parser has already completed the first call. Release an ordered prefix of parser-completed calls so clients can consume the first tool while later calls are still generating.

Key Changes

61 added nonblank, noncomment production lines in the Messages converter, excluding tests and documentation.

• Consume existing per-call completion evidence, including completion-only chunks and identity or arguments that arrive later. Require complete identity and valid object input before early emission; a pending earlier call remains an ordering barrier.
• Preserve the existing finish/EOF fallback for streams without completion evidence, the single-tool limit, per-fragment usage, and duplicate suppression. Empty argument strings wait until EOF even with completion evidence; explicit {} can release early. Warn once per call if non-empty arguments arrive after emission or finalization, without logging their contents. No request or worker wire fields change.
• Extend the existing gated HTTP replay and document the Messages streaming behavior.

Validation

A request such as the following can produce two read_file calls:

{
  "model": "MODEL",
  "max_tokens": 128,
  "stream": true,
  "tools": [{
    "name": "read_file",
    "input_schema": {
      "type": "object",
      "properties": {"path": {"type": "string"}},
      "required": ["path"]
    }
  }],
  "messages": [{"role": "user", "content": "Read /a and /b"}]
}

Case	Before	With this change / evidence
First call has complete=true; second call and turn finish are held behind a gate	No first tool block reaches the client	completed_tool_block_precedes_gated_tail observes content_block_stop through a real reqwest stream before releasing the existing semaphore gate, then receives the second block without duplication
A later call completes before an earlier call's arguments or identity	No per-call release	parser_completion_waits_for_ordered_valid_calls_and_late_identity preserves call order through metadata-only and repeated completion, invalid-argument barriers, late identity, and the original fragment usage snapshots
Empty argument string with completion evidence and disable_parallel_tool_use=true	Name-only parser children can close a block before arguments arrive	parser_completion_keeps_legacy_fallback_and_single_tool_limit waits until EOF and preserves the single-tool limit
Real V2 adapter fan-out, including DeepSeek V4 name-only children	The previous condition could publish {} and discard later arguments	v2_parser_arguments_survive_anthropic_conversion_at_every_split feeds native and named guided calls through apply_stream and the Messages converter for 12 families at every UTF-8 split; the old condition fails this regression

The unchanged-parent converter fails the first-call-before-finish assertion in a source-isolated replay; the modified converter emits the first call's three SSE events immediately. Native CPU validation passes all 118 Anthropic unit tests and 15 HTTP replay tests. The complete HTTP replay suite takes 2.78 seconds on macOS arm64 with default features disabled.

The HTTP backend supplies completion evidence explicitly to qualify early HTTP delivery. The new CPU adapter-to-converter regression additionally covers real V2 parser output for Qwen3, Kimi K2/K3, DeepSeek V4/V4.1, Gemma4, GLM4.7, Muse Glimmer/Spark, Nemotron v3, GPT-OSS, and Beam. Early release requires V2 completion evidence; V1 paths retain the fallback. This does not qualify live-model serving. GPU validation and latency/throughput benchmarking remain unrun; no speedup percentage is claimed.

Issues and Ongoing Work

Closes #17748.

• feat(llm): stream tool calls with the V2 parser adapter #15577 introduced the existing per-call parser completion signal; the Messages converter now consumes it.
• Merged fix(frontend): complete anthropic message api's tool calls after early finish #15129 handles delayed arguments after an early turn finish. This change additionally releases calls before a turn finish is received.
• fix(frontend): stop tool_call_dispatch handing harnesses truncated arguments #13986 implements the Chat Completions dispatch extension; this PR changes standard Messages tool blocks.
• fix(anthropic): preserve tool failures and thinking display #17381 changes thinking/tool ordering in the same converter. It does not consume per-call completion evidence; coordinate the shared-file changes when combining the PRs.
• Checked current main and paginated all 1,508 open PRs, including 424 drafts. PR searches for tool_call_completion, Anthropic completed tools, and stream_converter found no implementation of this Messages behavior. fix(preprocessor): finalize every reasoning parser at end of stream #14934 changes reasoning-parser EOF finalization, not the Messages converter.

Pull Request Checklist

• I used Dynamo's /pr-checklist skill, including independent design, testing, and code-quality/performance reviews.

Summary by CodeRabbit

• Bug Fixes
  • Streaming responses now release completed tool calls in their original order as soon as completion is confirmed and the call details are valid. Later calls wait until earlier calls are complete, preventing calls from appearing out of order.
  • Tool-call finish signals continue to trigger a flush but do not by themselves mark calls complete. Calls with missing or incomplete arguments remain pending during streaming.
• Documentation
  • Updated the Anthropic API configuration reference to describe tool-call streaming behavior.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[dynamo-ops]

/ok to test fc024e7

[coderabbitai]

No actionable comments were generated in the recent review.

Recent review info

Run configuration

• Source mode: Auto
• Repository: ai-dynamo/dynamo
• Branch: main
• File: .coderabbit.yaml
• Commit: cb5f9a6
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: abe9611f-7da3-4c85-82c7-5a7f0895678f

Commits

Reviewing files that changed from the base of the PR and between cb5f9a6 and fc024e7.

Files selected for processing: 3

• docs/fern/pages/reference/components/frontend-configuration.mdx
• lib/llm/src/protocols/anthropic/stream_converter.rs
• lib/llm/tests/anthropic_http_replay.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

The Anthropic stream converter now uses per-call parser completion evidence to release ready tool calls before the turn finishes. It preserves ordered emission and finish or end-of-stream fallback behavior. Tests and reference documentation cover the updated streaming behavior.

Changes

Tool call streaming

Layer / File(s)	Summary
Track completion and flush calls in order lib/llm/src/protocols/anthropic/stream_converter.rs, lib/llm/tests/anthropic_http_replay.rs, docs/fern/pages/reference/components/frontend-configuration.mdx	The converter records completion per call and emits only an ordered prefix of ready calls during completion-driven drains. Finish-triggered and final drains retain fallback behavior. Tests cover completion ordering, delayed identity, fragmented output, finish handling, and single-tool mode. The documentation describes the release conditions and fallback.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Severity of issue fixed: Medium

Merge Risk: Minimal · up to fc024

The reviewed streaming change has no identified issue requiring resolution before merge.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title clearly identifies the main change: flushing parser-completed Anthropic tool calls before turn completion.
Linked Issues check		Directly linked issue #17748 is open and active, so its coding requirements apply. Requirements and evidence at head fc024e7: 1. [#17748] Consume existing per-call completion evidence. `process_chun…
Out of Scope Changes check		The whole-PR diff from merge base cb5f9a6 to head fc024e7 touches three code or doc areas, and each connects to #17748: - lib/llm/src/protocols/anthropic/stream_converter.rs: the core change. It …
Description check		The pull request description includes all required sections: Overview, Key Changes, Validation, Issues and Ongoing Work, issue closure, and the checklist. It provides detailed scope, behavior, testing…

Full details: Linked Issues check

Explanation

Directly linked issue #17748 is open and active, so its coding requirements apply. Requirements and evidence at head fc024e7:

1. [#17748] Consume existing per-call completion evidence. process_chunk now reads chunk.tool_call_completion for choice 0 and sets is_complete and needs_flush on the indexed, un-emitted call. record_tool_call returns is_complete, so late identity can trigger a flush. Verified in the diff.
2. [#17748] Release only an ordered, complete prefix. In drain_buffered_tool_events, when only_completed is true, the loop breaks at the first call that is not complete, not emit-ready, or not flush-pending. Calls are not released out of order. Verified in the diff.
3. [#17748] Require complete identity and valid object input before early emission. The only_completed path requires is_emit_ready() (backend_id and name non-empty). The repair path is gated by is_final, so repaired input is never used early. Invalid or incomplete JSON breaks the loop. serde_json::Map parsing rejects non-object input. Verified in the diff.
4. [#17748] Keep finish/EOF fallback when no evidence exists. only_completed is false once tool_flush_requested is set, and the non-completed path is unchanged. The legacy-fallback test asserts this.
5. [#17748] Preserve single-tool limits, per-fragment usage, and duplicate suppression. The single_tool && has_emitted break, per-fragment usage_snapshot retention, and has_emitted skip are retained. The test asserts usage values [1, 3, 2].
6. [#17748] Tests. parser_completion_waits_for_ordered_valid_calls_and_late_identity covers metadata-only completion, repeated completion, late identity, and an incomplete-argument barrier. parser_completion_keeps_legacy_fallback_and_single_tool_limit covers the parameterless single-tool case and the no-evidence fallback. completed_tool_block_precedes_gated_tail (parameterized) uses the existing gate in the HTTP replay harness, with a two-second drain timeout, to check that the first content_block_stop arrives before the tail is released.

Unverified points: I did not run tests. I did not inspect the parallel-tools.sse fixture contents. The issue asks for error-path coverage. The diff adds no explicit terminal-error test. The incomplete-argument path is covered, and error handling is unchanged by this diff, so this is not treated as an unmet requirement. Completion evidence for a tool index with no recorded state is ignored. The parser emits call chunks before completion, so this is not shown to be a gap.

Full details: Out of Scope Changes check

Explanation

The whole-PR diff from merge base cb5f9a6 to head fc024e7 touches three code or doc areas, and each connects to #17748:

• lib/llm/src/protocols/anthropic/stream_converter.rs: the core change. It adds is_complete state, completion handling in process_chunk, the ordered only_completed barrier, and the new tests. All hunks serve the early per-call release objective or its tests.
• lib/llm/tests/anthropic_http_replay.rs: replaces the single usage-tail test with the parameterized completed_tool_block_precedes_gated_tail, adds a per-tool completion case, and adds a two-second drain timeout. These changes support the HTTP replay validation that the issue requests.
• docs/fern/pages/reference/components/frontend-configuration.mdx: adds five lines that document the Messages streaming release rule. Documentation of the changed behavior is a legitimate supporting change.

No change in the reviewed diff addresses unrelated behavior. No request or worker wire fields, public declarations, or dependencies changed, per the summaries. The ignored_files list is empty.

Full details: Description check

Explanation

The pull request description includes all required sections: Overview, Key Changes, Validation, Issues and Ongoing Work, issue closure, and the checklist. It provides detailed scope, behavior, testing results, limitations, and related work.

Full details: Docstring Coverage

Explanation

Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 6 functions across 2 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a199e4-17b8-75af-ba07-85b99bf183f0.docs.buildwithfern.com/dynamo/dev

[devin-ai-integration]

✅ Dynamo PR CI passed — run 41372691351 (attempt 1) on c0f905d351

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

_{Posted automatically by Devin for run 41372691351. Updated on every full-CI run of this PR.}

[dynamo-ops]

/ok to test c0f905d

[rmccorm4]

Approving docs change