fix(grpc): report max_tokens and skip MCP dispatch for tool calls cut short

[yechank-nvidia]

Description

Problem

A streaming tool parser reports a call's name as soon as it is complete and the arguments after it. When the output stops inside a call, or a call's arguments never become JSON, two gRPC paths still treat the call as a finished one.

Messages reports tool_use for a truncated response. The stop reason is tool_use whenever a call was parsed, even when the engine stopped at the token limit. With qwen_xml, an output cut at max_tokens inside a call (<tool_call>\n<function=lookup>\n<parameter=city>\nPar) starts a tool_use block and ends with stop_reason: "tool_use", so a client that acts on tool_use runs the tool with a partial input. A whole call followed by text that is cut at the limit ends the same way, streamed or not. For the same output, Chat reports finish_reason: "length" and Responses reports incomplete.

The Responses tool loop runs an MCP call whose arguments are not JSON. It runs the server tool with {}, as it does for any arguments that are not a JSON object. When streaming, a server call reaches the loop with arguments that are not JSON when the output stops (finish_reason: "stop"):

• after the call's name: json, llama, mistral, qwen (<tool_call>\n{"name": "brave_web_search") and inkling;
• inside the arguments: deepseek, deepseek31, kimik2, hy_v4 and inkling ({"city": "Par);
• after a complete call whose arguments are not valid JSON: deepseek, deepseek31 and kimik2 ({"city": Paris}).

The non-streaming parse of these outputs returns no call. In the contract-test harness, where the mock model repeats its output, main runs the tool with {} on every turn of the streamed response until the internal limit of 10 calls, and the response fails with max_tool_calls_exceeded. The same request without streaming completes and runs nothing.

A guard on the MCP loop needs a parser fix first. json, llama, mistral, qwen and cohere (through the shared JSON streaming helper) and minimax_m2 stream only the name of a complete call that takes no arguments, such as {"name": "ping"} or an <invoke> without parameters. Their non-streaming parse returns {} for it. A Chat stream then ends with the call's arguments missing, and with the MCP guard alone a streamed MCP call without arguments would no longer run. The JSON helper also stays on such a call, so a later call in the same stream is lost: json and qwen stream one call for two calls without arguments.

Solution

Three commits, the parser fix first:

1. fix(tool_parser): stream {} when a call without arguments completes, as qwen_xml already does. The JSON helper then moves on to the next call, as after a call with arguments. A call cut short before its arguments still has none.
2. fix(grpc): in Messages, check a length finish before the parsed calls and report max_tokens, in the streaming and the non-streaming path. The content blocks are unchanged.
3. fix(responses): treat a server call whose arguments are not JSON like the calls of a truncated or failed generation, in the streaming and the non-streaming loop. Nothing in that batch is dispatched, the loop ends, and the unfinished server call is not handed to the client as a function call. Results of earlier iterations are kept. Arguments that are JSON but not an object still run with {}, and so does a complete call without arguments, which now streams {}. The non-streaming parse of every registered parser returns JSON arguments, so there the check only keeps the two loops alike.

Changes

• crates/tool_parser/src/parsers/helpers.rs: in handle_json_tool_streaming, a complete call without arguments (or parameters) takes {}.
• crates/tool_parser/src/parsers/minimax_m2.rs: at </invoke>, stream {} when nothing was streamed for the call.
• crates/tool_parser/tests/
  • tool_parser_streaming_flush.rs: a_call_without_arguments_streams_an_empty_object for json, llama, mistral and qwen, fed in one chunk and in 2–3 character chunks.
  • tool_parser_minimax_m2.rs: test_minimax_streaming_empty_parameters.
  • tool_parser_cohere.rs: test_cohere_streaming_empty_parameters.
• model_gateway/src/routers/grpc/regular/streaming.rs and processor.rs: the Messages stop_reason checks length first.
• model_gateway/src/routers/grpc/regular/streaming/eof_tests.rs: messages_truncated_after_a_tool_call_starts_stop_at_max_tokens runs qwen_xml through the streaming and the non-streaming Messages path, with length and stop.
• model_gateway/src/routers/grpc/regular/responses/common.rs: has_unfinished_mcp_call. The streaming and the non-streaming loop call it next to the existing check for a truncated or failed generation.
• model_gateway/tests/grpc_responses_stream_contract_test.rs
  • mcp_call_without_json_arguments_never_dispatches: the output ends after the server call's name.
  • mcp_call_without_arguments_runs_with_an_empty_object: a complete server call without arguments still runs, with {}.

The harmony Responses loop is not changed.

Behavior changes

• Messages: max_tokens instead of tool_use when the finish is length and a call was parsed, including a named tool_choice. A stop finish with calls still gives tool_use. Chat and Responses are unchanged.
• Responses with MCP tools: a batch that holds a server call whose arguments are not JSON, including an empty string, is not run, and neither are the other server calls in that batch. The response status comes from the finish, so a stop finish gives completed. Function tools and arguments that are JSON but not an object are unchanged.
• Streaming tool calls: a complete call without arguments now streams {} with json, llama, mistral, qwen, cohere and minimax_m2, as their non-streaming parse returns.

Test Plan

Negative control. I ran every new test by name on main (61b250f7) with only the test files of this PR added.

Test	main	This PR
a_call_without_arguments_streams_an_empty_object	fail: [json] arguments "" instead of {}	pass
test_minimax_streaming_empty_parameters	fail: "" instead of {}	pass
test_cohere_streaming_empty_parameters	fail: "" instead of {}	pass
messages_truncated_after_a_tool_call_starts_stop_at_max_tokens	fail: tool_use instead of max_tokens (streamed, cut inside the call)	pass
mcp_call_without_json_arguments_never_dispatches	fail: streamed response failed after 10 MCP calls	pass
mcp_call_without_arguments_runs_with_an_empty_object	pass (main runs every call)	pass

The last test guards the parser fix together with the MCP guard. With the MCP guard and without the parser fix, its streamed case fails (see the mutation check).

Mutation check. On a copy of this branch, I reverted one change at a time and ran the new tests.

Reverted	Failing test
the parser fix, keeping the MCP check	mcp_call_without_arguments_runs_with_an_empty_object: the streamed call runs 0 times instead of once
the check in the streaming MCP loop	mcp_call_without_json_arguments_never_dispatches: the streamed response fails
the check in the non-streaming MCP loop	none, since no registered parser returns arguments that are not JSON without streaming
the stop_reason order in Messages streaming	messages_truncated_after_a_tool_call_starts_stop_at_max_tokens: tool_use for the streamed call cut inside
the stop_reason order in Messages non-streaming	messages_truncated_after_a_tool_call_starts_stop_at_max_tokens: tool_use for the whole call followed by text

All registered tool parsers. A scratch test fed a complete call without arguments to each of the 24 registered parsers other than passthrough, token by token with special tokens kept whole, and compared the streamed arguments with the non-streaming parse. On main, json, llama, mistral, qwen, cohere and minimax_m2 stream "" where the non-streaming parse returns {}. With this PR, all 24 stream the same arguments as their non-streaming parse. The same scratch test fed the whole output in one chunk. The differences left there are existing ones that do not depend on arguments: json, mistral and qwen stream only the first of two calls given in one chunk (mistral token by token too), and cohere and step3 stream no call when the whole block comes in one final chunk.

Commands. All ran offline on CPU.

Check	Result
cargo +nightly fmt --all -- --check	pass
cargo test -p tool-parser	550 passed (3 new)
cargo test -p smg --lib	2,031 passed (1 new, in eof_tests)
cargo test -p smg --test grpc_responses_stream_contract_test / messages_streaming_test / messages_test / api_tests / spec_test / grpc_context_length_test / grpc_pd_fanout_test	31 (2 new) / 21 / 11 / 109 / 98 / 17 / 13 passed
cargo clippy --workspace --all-targets -- -D warnings; -p smg --all-targets with default features and with grpc-server,jemalloc-profiling,test-util; CI's two --no-default-features variants; -p tool-parser --all-targets --all-features	pass
Each commit on its own: fmt, workspace clippy, tool-parser tests, eof_tests, grpc_responses_stream_contract_test	pass
pre-commit run --all-files with CI's SKIP list, and the commit-msg hooks on each commit	pass

Open PRs. A 3-way merge with #2715 and with #2716 is clean in both orders. On the merge of this PR with each of them, eof_tests, grpc_responses_stream_contract_test, messages_streaming_test, messages_test and cargo clippy -p smg --all-targets -- -D warnings pass. The new Messages test builds its MessagesResponseSpec from a request, so it does not need the field that #2716 adds to that struct.

The offline crate cache did not have the three dependency bumps on main: lru 0.18.5, cc 1.5.1 and the opentelemetry-proto 0.33 dev-dependency. So the build copies used the Cargo.lock from before those bumps and the dev-dependency at 0.32. No source file differed.

Checklist

• cargo +nightly fmt passes
• cargo clippy --all-targets -- -D warnings passes for the workspace, -p smg with default features and with grpc-server,jemalloc-profiling,test-util, CI's two --no-default-features variants, and -p tool-parser --all-features. --all-features for the workspace needs OpenCV and was not run offline.
• (Optional) Documentation updated: not needed, no option or API changed
• (Optional) Please join us on Slack #sig-smg to discuss, review, and merge PRs

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: ce7f7e8c-dcb7-40d5-ab53-b94ee691fdf2

📥 Commits

Reviewing files that changed from the base of the PR and between eb39922 and 16cf7c3.

📒 Files selected for processing (12)

• crates/tool_parser/src/parsers/helpers.rs
• crates/tool_parser/src/parsers/minimax_m2.rs
• crates/tool_parser/tests/tool_parser_cohere.rs
• crates/tool_parser/tests/tool_parser_minimax_m2.rs
• crates/tool_parser/tests/tool_parser_streaming_flush.rs
• model_gateway/src/routers/grpc/regular/processor.rs
• model_gateway/src/routers/grpc/regular/responses/common.rs
• model_gateway/src/routers/grpc/regular/responses/non_streaming.rs
• model_gateway/src/routers/grpc/regular/responses/streaming.rs
• model_gateway/src/routers/grpc/regular/streaming.rs
• model_gateway/src/routers/grpc/regular/streaming/eof_tests.rs
• model_gateway/tests/grpc_responses_stream_contract_test.rs

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 2 remain after this review.

---

📝 Summary

Summary by CodeRabbit

• Bug Fixes
  • Completed tool calls that omit arguments now provide an empty JSON object ({}) consistently across supported formats and streaming modes. Incomplete calls remain undispatched.
  • MCP calls with incomplete arguments are no longer executed or exposed as client function calls.
  • Responses truncated by token limits now report MaxTokens, even when tool calls are present.

Walkthrough

Complete tool calls without arguments now emit {}. The gateway detects unfinished MCP calls before dispatch and selects MaxTokens when a Messages completion ends with the length finish reason.

Changes

Empty tool-call arguments

Layer / File(s)	Summary
Emit empty arguments for complete calls crates/tool_parser/src/parsers/helpers.rs, crates/tool_parser/src/parsers/minimax_m2.rs, crates/tool_parser/tests/tool_parser_cohere.rs, crates/tool_parser/tests/tool_parser_minimax_m2.rs, crates/tool_parser/tests/tool_parser_streaming_flush.rs	Complete calls without arguments emit {}. Tests cover single-chunk and split inputs across the Cohere, Minimax M2, JSON, Llama, Mistral, and Qwen parsers.

Gateway tool-call response handling

Layer / File(s)	Summary
Check MCP calls before dispatch model_gateway/src/routers/grpc/regular/responses/common.rs, model_gateway/src/routers/grpc/regular/responses/non_streaming.rs, model_gateway/src/routers/grpc/regular/responses/streaming.rs, model_gateway/tests/grpc_responses_stream_contract_test.rs	Both MCP tool loops check for exposed tool calls with arguments that fail JSON parsing before dispatch. Tests cover incomplete calls and complete calls without an arguments field.
Select stop reason for length completions model_gateway/src/routers/grpc/regular/processor.rs, model_gateway/src/routers/grpc/regular/streaming.rs, model_gateway/src/routers/grpc/regular/streaming/eof_tests.rs	Messages processing selects MaxTokens when the engine finish reason is length, including when tool calls are present. Tests cover length and stop finish reasons.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 16cf7

The change preserves empty arguments for complete tool calls, blocks incomplete MCP dispatch, and reports max_tokens for length-limited Messages responses. No merge-blocking issue remains identified; merge after normal checks.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 16cf7

The change reduces accidental execution of incomplete tool calls and reports truncation more accurately. No introduced security issue was established, but downstream client behavior and production security boundaries are not fully evidenced.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — Model-controlled names and arguments can reach tools in the existing session exposure map and execute using their bound server authority. The inspected change narrows execution eligibility without enlarging that map. The visible map is not proof of downstream tenant isolation or limited remote privileges.

Security Findings and Attack Paths

• observed — Previously, malformed server-call arguments could reach dispatch preparation and become {}. The new pre-dispatch guard blocks that path in both response modes. Inspected contract assertions distinguish an incomplete named call, which must execute zero times, from a complete omitted-argument call, which must execute with {}.

Trust Boundaries and Controls

• observed — The guard and dispatch partition use the same session-exposed identity test. Existing session construction applies selected-server tool allowlists, and execution resolves the exposed name to a bound server and approval context. The completeness check supplements these controls; it does not replace authorization.

Resilience and Maintainability Implications

• observed — Existing execution budgets bound repeated server calls, and the new guard terminates malformed batches before consuming that execution allowance. Base/head comparison also confirms that valid mixed server/client execution and suppression of server-call deltas from client-function events predate this PR.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the primary gRPC changes: reporting max_tokens for truncated tool calls and skipping MCP dispatch for incomplete calls.
Description check	✅ Passed	The description is directly related to the changeset and provides detailed problem, solution, behavior, and test information for parser, Messages, and Responses changes.
Docstring Coverage	✅ Passed	Docstring coverage is 96.15% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 26 functions across 12 files.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

_{Comment @coderabbitai help to get the list of available commands.}