Repository navigation
[Bugfix] Stop Kimi K3 end-of-turn marker leaking into streamed content - #51400
stantheman0128 wants to merge 1 commit into
Conversation
The model closes the assistant turn with <|close|>message<|sep|> right before the end-of-message token. The non-streaming path strips it in _strip_response_content, but _extract_response_content only stopped content emission at the tools-open and response-close markers, so a turn that ends without a response-close marker streamed the raw XTML marker to the client: streaming returned 'Hello<|close|>message<|sep|>' where extract_tool_calls returns 'Hello'. Treat a complete <|close|>message<|sep|> as a content stop the same way as response-close, and hold back a partial marker suffix at the end of the accumulated text so a marker split across deltas cannot leak either. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: stantheman0128 <stanshih888@gmail.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment Once the PR is approved or has the If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
This branch has not been deployed
2 participants
FIX #51399
Purpose
The model closes a Kimi K3 assistant turn with
<|close|>message<|sep|>right before the end-of-message token (see the_K3_MESSAGE_CLOSEcomment instructural_tag_registry.py). The non-streaming tool parser strips that marker in_strip_response_content, but the streaming content path_extract_response_contentonly stopped content emission attools openandresponse close. A turn that ends without aresponse closemarker therefore streamed the raw XTML marker to the client:| Path | Content for
Hello<|close|>message<|sep|>|| --- | --- |
|
extract_tool_calls(non-streaming) |Hello||
extract_tool_calls_streamingbefore this PR |Hello<|close|>message<|sep|>||
extract_tool_calls_streamingafter this PR |Hello|The fix treats a complete
<|close|>message<|sep|>as a content stop the same wayresponse closealready is, and adds the marker to the partial-suffix hold-back so a marker split across deltas cannot leak either. Behavior for content that ends in a partial marker prefix at truncation matches how the existingresponse_closehold-back already behaves.Level of fix: the unary strip set is
response_open,response_close,message_close; after this change the streaming stop-and-hold set covers exactly that set, so the divergence class inside this parser is closed. The reasoning parser keeps its own already-correct copy of this logic; unifying the two into a shared helper would couple the reasoning and tool parsers and is out of scope for a bugfix.Why this is not duplicating an existing PR
Open Kimi K3 tool parser PRs #50627 and #50628 make tool-call argument streaming incremental (issue #50604). Neither touches
_extract_response_contentor the content marker logic (checked their diffs on 2026-08-07). No open issue or PR covers the content marker leak.Test Plan
New
tests/tool_parsers/test_kimi_k3_tool_parser.pywith two behavior tests:test_streaming_content_stops_at_message_close: streaming equals non-streaming for a turn ending in<|close|>message<|sep|>withoutresponse close.test_streaming_content_holds_split_message_close: the marker never leaks when deltas split it at chunk sizes 1, 3, and 7.Test Result
Both new tests fail before the fix (
AssertionError: assert 'Hello<|close|>message<|sep|>' == 'Hello') and pass after it. The pre-existingtest_kimi_k3_named_tool_choice.pystill passes. A local parity harness (7 output shapes including full response wrapper, consumed prefix, single and double tool calls, unicode and empty-string arguments, object arguments, each split at chunk sizes 1/3/7/full) reports zero streaming vs non-streaming mismatches after the fix.This change affects serving-layer string extraction only, no model forward path, so no model evaluation is attached; the parity harness against
extract_tool_callscovers the output contract.An independent adversarial verification pass (separate agent, fresh context) confirmed the bug on the unfixed tree with its own fuzzer (the marker leaks at all 27 split offsets before the fix, at none after), confirmed reachability (
vllm/parser/kimi_k3.pylines 116-119 bypass the reasoning-parser stripping whenever a tool parser is configured), and measured the one behavior change: content ending in a partialmessage_closeprefix at EOS is now held back up to 22 chars, the same class of hold the existingresponse_closeguard has always applied at up to 23 chars.What was not tested
No end-to-end serving run with a real Kimi K3 model: verification is parser-level on a CPU box, so the claim that a real detokenizer delivers the marker as contiguous literal text rests on
adjust_requestforcingskip_special_tokens=Falseandspaces_between_special_tokens=False, not on a live run. Whitespace-tolerant marker variants (<|close|> message <|sep|>) still leak on both trees; that gap is pre-existing forresponse_closetoo, is documented in the parser as defense-in-depth made a no-op byadjust_request, and stays out of scope here.AI assistance disclosure
This bug was found and fixed with AI assistance (Claude). The submitting human reviewed every changed line and ran the tests locally.