fix(claude): emit tool args before deferring stream close on finish_reason

[denghuinow]

Summary

• Fix intermittent truncated input_json_delta streams when converting OpenAI-compatible upstream tool-call SSE to Claude Messages format.
• When upstream sends finish_reason without usage in the same chunk, the converter now still emits any tool_calls argument deltas from that chunk before deferring message_stop.
• Add regression tests covering the finish_reason → trailing usage chunk sequence.

Related issues

• Partially addresses fix: Claude 協議下工具調用 (tool_use) 出現空 input {} 或數據截斷問題 #4203 (root cause 3: early return on finish_reason without usage drops the last tool_calls argument delta). Causes 1–2 in that issue (duplicate content_block_start, pointer aliasing) are not covered by this PR.
• Related to 通过CC调用GLM4.7-flash，涉及工具调用时出错 #4379 (Claude Code → new-api → GLM/vLLM tool failures such as Invalid tool parameters / Error writing file; likely the same OpenAI⇄Claude stream conversion path).

Problem

Some OpenAI-compatible upstreams (e.g. GLM) emit streaming chunks in this order:

1. multiple tool_calls[].function.arguments fragments
2. a chunk with finish_reason: tool_calls (often containing the final argument fragment, without usage)
3. a final usage-only chunk

Previously, step 2 triggered an early return before processing tool_calls in the same chunk, so the final partial_json fragments were dropped and clients saw invalid JSON like:

{"command": "ls -la /root 2>/dev/null | head -20", "description": "列出 /root 目录下的文件（前20行）

Test plan

• go test ./service/ -run TestStreamResponseOpenAI2Claude
• Manual Claude Messages streaming tool-call loop (20/20 OK after fix)

Manual reproduction (20-run loop)

Set BASE to your new-api endpoint, KEY to a valid API key, and MODEL to an OpenAI-compatible upstream that supports tool calls (e.g. glm-5.2).

cat >/tmp/check-claude-stream-loop.sh <<'SH'
#!/usr/bin/env bash
set -euo pipefail

BASE="${BASE:-http://127.0.0.1:3000}"
KEY="${KEY:?set KEY to your API key}"
MODEL="${MODEL:-glm-5.2}"

cat >/tmp/claude-tool-test.json <<JSON
{
  "model": "$MODEL",
  "max_tokens": 512,
  "stream": true,
  "tools": [
    {
      "name": "Bash",
      "description": "Run a bash command",
      "input_schema": {
        "type": "object",
        "properties": {
          "command": {"type": "string"},
          "description": {"type": "string"}
        },
        "required": ["command"]
      }
    }
  ],
  "tool_choice": {
    "type": "tool",
    "name": "Bash"
  },
  "messages": [
    {
      "role": "user",
      "content": "调用 Bash 执行：ls -la /root 2>/dev/null | head -20"
    }
  ]
}
JSON

ok=0
fail=0

for i in $(seq 1 20); do
  out="/tmp/claude-tool-test-$i.sse"

  curl -sS -N "$BASE/v1/messages?beta=true" \
    -H "x-api-key: $KEY" \
    -H "anthropic-version: 2023-06-01" \
    -H "Content-Type: application/json" \
    --data-binary @/tmp/claude-tool-test.json \
    > "$out"

  if python3 - "$out" <<'PY'
import json, sys

path = sys.argv[1]
s = ""

for line in open(path, encoding="utf-8"):
    line = line.rstrip("\n")
    if not line.startswith("data: "):
        continue
    data = line[6:].strip()
    if not data or data == "[DONE]":
        continue
    try:
        obj = json.loads(data)
    except Exception:
        continue
    if obj.get("type") == "content_block_delta":
        delta = obj.get("delta") or {}
        if delta.get("type") == "input_json_delta":
            s += delta.get("partial_json") or ""

if not s:
    print("FAIL empty partial_json")
    sys.exit(1)

try:
    json.loads(s)
    print("OK", s)
except Exception as e:
    print("FAIL", repr(e), s)
    sys.exit(1)
PY
  then ok=$((ok+1)); else fail=$((fail+1)); fi
done

echo "summary: ok=$ok fail=$fail"
SH

chmod +x /tmp/check-claude-stream-loop.sh
KEY=sk-your-api-key BASE=http://127.0.0.1:3000 MODEL=glm-5.2 /tmp/check-claude-stream-loop.sh

Expected after fix: summary: ok=20 fail=0

Typical failure before fix: intermittent Unterminated string / Expecting ',' delimiter errors, e.g. summary: ok=9 fail=11

Made with Cursor

Summary by CodeRabbit

• Bug Fixes
  • Improved streaming responses when tool calls finish before usage information arrives.
  • Tool-call arguments now continue streaming correctly instead of being cut off prematurely.
  • Ensured arguments accumulated across multiple updates form valid JSON for reliable processing.
  • Final completion and usage details are emitted only after the required usage information is available.

[coderabbitai]

Walkthrough

StreamResponseOpenAI2Claude now defers final stream closure until usage is available, while preserving tool-argument emission. New tests cover finish-before-usage handling and valid JSON assembled from multiple tool-call deltas.

Changes

Claude stream finish/usage fix

Layer / File(s)	Summary
Finalization logic deferring message_delta until usage present service/relayconvert/internal/oai_chat/to_claude_messages_resp.go	Finalization now waits for usage data before stopping open blocks and emitting the final stop-reason delta.
Tool-argument streaming validation service/relayconvert/internal/oai_chat/to_claude_messages_resp_test.go	Tests verify tool arguments are emitted before usage and that accumulated JSON fragments form valid JSON.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• QuantumNous/new-api#4090: Modifies the same finalization path for usage-only chunks and final Claude stop emission.
• QuantumNous/new-api#2397: Adjusts tool-call argument handling in StreamResponseOpenAI2Claude.
• QuantumNous/new-api#1531: Changes tool-call handling in the OpenAI-to-Claude streaming converter.

Suggested reviewers: calcium-ion

Sequence Diagram(s)

sequenceDiagram
  participant OpenAIStream
  participant StreamResponseOpenAI2Claude
  participant ClaudeStream
  OpenAIStream->>StreamResponseOpenAI2Claude: tool_calls finish chunk without usage
  StreamResponseOpenAI2Claude->>ClaudeStream: emit input_json_delta
  StreamResponseOpenAI2Claude-->>ClaudeStream: defer final message_delta
  OpenAIStream->>StreamResponseOpenAI2Claude: usage chunk
  StreamResponseOpenAI2Claude->>ClaudeStream: emit final message_delta and stop

Loading

Poem

I’m a rabbit hopping through the stream,
Joining JSON bits into one bright dream.
Tool calls bloom before the stop,
Usage arrives, then closures pop.
Thump-thump! The Claude message is complete.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main fix: emitting tool arguments before stream close is deferred on finish_reason.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@service/relayconvert/internal/oai_chat/to_claude_messages_resp.go`:
- Around line 241-263: In the first-chunk finalization branch of the response
conversion flow, defer closing until usage is available: when oaiUsage is nil,
return without calling stopOpenBlocks(), emitting message_stop, or setting
ClaudeConvertInfo.Done. Only perform finalization after usage is present,
matching the multi-chunk path, and add a regression test covering
SendResponseCount == 1 with a later usage-only chunk.
- Around line 402-425: Add an EOF fallback in the done-chunk handling around
`oaiUsage` so the direct `ConvertStream` path always emits the final Claude
`message_delta` and `message_stop` when no usage chunk arrives. Do not return
solely because `oaiUsage` is nil; emit the completion using a zero or otherwise
established fallback usage, while preserving the existing upstream usage
selection and `info.ClaudeConvertInfo.Done` behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 776fb58d-13a8-4bde-8f06-1a75fc10d049

📥 Commits

Reviewing files that changed from the base of the PR and between 5251a1a and 26481b6.

📒 Files selected for processing (2)

• service/relayconvert/internal/oai_chat/to_claude_messages_resp.go
• service/relayconvert/internal/oai_chat/to_claude_messages_resp_test.go

[coderabbitai]

Caution

Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@service/relayconvert/internal/oai_chat/to_claude_messages_resp.go`:
- Around line 241-263: In the first-chunk finalization branch of the response
conversion flow, defer closing until usage is available: when oaiUsage is nil,
return without calling stopOpenBlocks(), emitting message_stop, or setting
ClaudeConvertInfo.Done. Only perform finalization after usage is present,
matching the multi-chunk path, and add a regression test covering
SendResponseCount == 1 with a later usage-only chunk.
- Around line 402-425: Add an EOF fallback in the done-chunk handling around
`oaiUsage` so the direct `ConvertStream` path always emits the final Claude
`message_delta` and `message_stop` when no usage chunk arrives. Do not return
solely because `oaiUsage` is nil; emit the completion using a zero or otherwise
established fallback usage, while preserving the existing upstream usage
selection and `info.ClaudeConvertInfo.Done` behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 776fb58d-13a8-4bde-8f06-1a75fc10d049

📥 Commits

Reviewing files that changed from the base of the PR and between 5251a1a and 26481b6.

📒 Files selected for processing (2)

• service/relayconvert/internal/oai_chat/to_claude_messages_resp.go
• service/relayconvert/internal/oai_chat/to_claude_messages_resp_test.go

🛑 Comments failed to post (2)

service/relayconvert/internal/oai_chat/to_claude_messages_resp.go (2)

241-263: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Locate the file and inspect the relevant section with line numbers.
file="service/relayconvert/internal/oai_chat/to_claude_messages_resp.go"
wc -l "$file"
sed -n '1,140p' "$file"
printf '\n--- SNIP ---\n'
sed -n '140,320p' "$file"
printf '\n--- TEST SEARCH ---\n'
rg -n "FinishReasonBeforeUsage|SendResponseCount == 1|message_stop|Done = true|Usage-only|usage-only" service/relayconvert/internal/oai_chat -n

Repository: QuantumNous/new-api

Length of output: 13509

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

file="service/relayconvert/internal/oai_chat/to_claude_messages_resp.go"
testfile="service/relayconvert/internal/oai_chat/to_claude_messages_resp_test.go"

printf '%s\n' '--- FINALIZATION PATH AROUND LINES 392-430 ---'
sed -n '392,430p' "$file"

printf '\n%s\n' '--- USAGE ASSIGNMENTS ---'
rg -n "ClaudeConvertInfo\.Usage|Usage =" "$file" "$testfile" service/relayconvert/internal/oai_chat -n

printf '\n%s\n' '--- RELEVANT TESTS ---'
sed -n '180,310p' "$testfile"

Repository: QuantumNous/new-api

Length of output: 8610

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

file="service/relayconvert/internal/oai_chat/to_claude_messages_resp.go"

printf '%s\n' '--- WHERE ClaudeConvertInfo.Usage IS UPDATED ---'
rg -n "ClaudeConvertInfo\.Usage\s*=|GetOrCreate.*Usage|Usage\)" "$file" service/relayconvert/internal/oai_chat -n

printf '\n%s\n' '--- BOTTOM OF FILE AROUND USAGE HANDLING ---'
sed -n '430,475p' "$file"

Repository: QuantumNous/new-api

Length of output: 7776

---

Defer first-chunk finalization until usage is available. When the very first chunk already has finish_reason but no usage, this branch still emits message_stop and sets Done, so a later usage-only chunk is dropped by the top-level guard and final billing data never reaches the client. Match the multi-chunk path here: if oaiUsage is nil, return without closing; only call stopOpenBlocks() and set Done once usage is present. Add a regression test for SendResponseCount == 1.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@service/relayconvert/internal/oai_chat/to_claude_messages_resp.go` around
lines 241 - 263, In the first-chunk finalization branch of the response
conversion flow, defer closing until usage is available: when oaiUsage is nil,
return without calling stopOpenBlocks(), emitting message_stop, or setting
ClaudeConvertInfo.Done. Only perform finalization after usage is present,
matching the multi-chunk path, and add a regression test covering
SendResponseCount == 1 with a later usage-only chunk.

---

402-425: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Find callers of StreamResponseOpenAI2Claude and any end-of-stream fallback logic.
rg -n "StreamResponseOpenAI2Claude" --type=go -B3 -A20 -g '!*_test.go'

# Check whether ClaudeConvertInfo.Usage is ever assigned.
rg -n "ClaudeConvertInfo\.Usage" --type=go

# Check whether upstream requests set stream_options.include_usage for OpenAI-compatible calls.
rg -n "include_usage" --type=go

Repository: QuantumNous/new-api

Length of output: 157

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Map the target file and inspect the relevant function.
ast-grep outline service/relayconvert/internal/oai_chat/to_claude_messages_resp.go --view expanded || true
sed -n '1,520p' service/relayconvert/internal/oai_chat/to_claude_messages_resp.go | nl -ba | sed -n '220,460p'

# Find callers and any end-of-stream fallback logic.
rg -n "StreamResponseOpenAI2Claude" --type=go -B3 -A20 -g '!*_test.go' .
rg -n "stopOpenBlocks|message_stop|message_delta|Done = true|ClaudeConvertInfo" --type=go -B3 -A8 -g '!*_test.go' service/relayconvert/internal/oai_chat

# Check whether ClaudeConvertInfo.Usage is assigned anywhere.
rg -n "ClaudeConvertInfo\.Usage|\.Usage\s*=" --type=go service/relayconvert/internal/oai_chat

# Check whether upstream requests set include_usage.
rg -n "include_usage|stream_options" --type=go .

Repository: QuantumNous/new-api

Length of output: 825

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Relevant section of service/relayconvert/internal/oai_chat/to_claude_messages_resp.go ==="
sed -n '220,460p' service/relayconvert/internal/oai_chat/to_claude_messages_resp.go

echo
echo "=== Callers of StreamResponseOpenAI2Claude and surrounding flow ==="
rg -n "StreamResponseOpenAI2Claude" --type=go -B4 -A20 -g '!*_test.go' .

echo
echo "=== Any end-of-stream / forced-close fallback logic in the relayconvert path ==="
rg -n "stopOpenBlocks|message_stop|message_delta|Done = true|Force.*close|synthetic.*message_stop|unterminated" --type=go -B4 -A10 service/relayconvert

echo
echo "=== Where ClaudeConvertInfo.Usage is assigned ==="
rg -n "ClaudeConvertInfo\.Usage|\.Usage\s*=" --type=go service/relayconvert

echo
echo "=== Where include_usage / stream_options are set for OpenAI-compatible requests ==="
rg -n "include_usage|stream_options" --type=go .

Repository: QuantumNous/new-api

Length of output: 50375

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Stream call site and surrounding EOF handling ==="
sed -n '850,930p' service/relayconvert/response_registry.go
rg -n "StreamResponseOpenAI2Claude|canonicalUsageFromResponse|EOF|end of stream|flush|final usage|usage-only" --type=go -B4 -A12 service/relayconvert

echo
echo "=== Exact assignments to ClaudeConvertInfo.Usage or related fields ==="
rg -n "ClaudeConvertInfo\.Usage\s*=|Usage\s*=\s*info\.ClaudeConvertInfo\.Usage|Usage\s*:=\s*.*ClaudeConvertInfo" --type=go service

echo
echo "=== OpenAI stream request construction and include_usage ==="
rg -n "include_usage|stream_options|StreamOptions|IncludeUsage" --type=go service

Repository: QuantumNous/new-api

Length of output: 46563

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Request-side stream options for OpenAI chat upstreams ==="
rg -n "IncludeUsage|include_usage|StreamOptions" --type=go service/relayconvert service

echo
echo "=== Stream finalization / EOF fallback in the response pipeline ==="
rg -n "Finalize.*Stream|finalize.*Stream|rememberUsage|Done = true|message_stop|message_delta" --type=go -B4 -A12 service/relayconvert/response_registry.go service/relayconvert/internal/oai_chat service/relayconvert/internal/oai_responses

echo
echo "=== ClaudeConvertInfo definition and any usage field writes outside this file ==="
rg -n "type ClaudeConvertInfo|Usage\s+\*dto\.Usage|ClaudeConvertInfo\.Usage" --type=go service

Repository: QuantumNous/new-api

Length of output: 50375

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Route registration for OpenAI chat -> Claude messages stream ==="
rg -n "OpenAI.*Claude|Claude.*OpenAI|FinalizeStream:|ConvertStream:" --type=go service/relayconvert/response_registry.go service/relayconvert -B4 -A10

echo
echo "=== ClaudeConvertInfo usage field definition and writes ==="
rg -n "type ClaudeConvertInfo|ClaudeConvertInfo\.Usage|Usage\s+\*dto\.Usage" --type=go service/relayconvert/internal service/relayconvert/response_registry.go -B3 -A6

echo
echo "=== Finalizer entrypoints used by response_registry ==="
sed -n '324,360p' service/relayconvert/response_registry.go

Repository: QuantumNous/new-api

Length of output: 50376

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Stream execution path for direct ConvertStream routes ==="
sed -n '470,620p' service/relayconvert/response_registry.go

echo
echo "=== OpenAI chat -> Claude messages route definition ==="
sed -n '48,80p' service/relayconvert/text_converter_registry.go

echo
echo "=== OpenAI chat request converter (to see whether include_usage is set) ==="
sed -n '1,220p' service/relayconvert/internal/oai_chat/to_claude_messages_req.go

Repository: QuantumNous/new-api

Length of output: 13029

---

Add an EOF fallback for the Claude stream. This direct ConvertStream route has no finalizer, and OpenAIChatRequestToClaudeMessages doesn’t set include_usage, so a provider that ends without a usage chunk will return from the oaiUsage == nil branch without ever sending message_delta/message_stop. info.ClaudeConvertInfo.Usage also isn’t populated in this path, so the fallback here won’t fire.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@service/relayconvert/internal/oai_chat/to_claude_messages_resp.go` around
lines 402 - 425, Add an EOF fallback in the done-chunk handling around
`oaiUsage` so the direct `ConvertStream` path always emits the final Claude
`message_delta` and `message_stop` when no usage chunk arrives. Do not return
solely because `oaiUsage` is nil; emit the completion using a zero or otherwise
established fallback usage, while preserving the existing upstream usage
selection and `info.ClaudeConvertInfo.Done` behavior.