Skip to content

fix(core): retry leaked JSON tool protocol output - #8301

Merged
wenshao merged 6 commits into
QwenLM:mainfrom
yiliang114:cx/fix-8207-protocol-json-leak
Aug 2, 2026
Merged

fix(core): retry leaked JSON tool protocol output#8301
wenshao merged 6 commits into
QwenLM:mainfrom
yiliang114:cx/fix-8207-protocol-json-leak

Conversation

@yiliang114

@yiliang114 yiliang114 commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

What this PR does

This PR prevents a model response that contains a JSON-array tool payload followed by leaked </parameter></function> protocol tags from reaching the UI, conversation history, or session recording. The failed attempt is routed through the existing protocol-leak retry path instead.

Detection stays at the shared stream boundary. It buffers a leading JSON object or object array without guessing argument names, preserves part ordering while the response is ambiguous, and releases ordinary JSON or real structured tool-call events unchanged. Numeric arrays and Markdown-style bracketed text continue streaming immediately.

Why it's needed

In a production session, the model returned finish_reason=stop with a plain-text JSON array containing two subagent argument objects and the closing tool protocol tags, but no structured tool call. The existing guard handled leading XML-style protocol tags, so this JSON-shaped variant was displayed and persisted as assistant text instead of being retried.

Reviewer Test Plan

How to verify

  1. Stream a thought plus a split JSON tool payload ending in </parameter></function>, both with a separate terminal event and with the finish reason on the content chunk. Cover an arbitrary first argument key, a direct object, and trailing prose. Confirm that the failed attempt emits no parts, writes no history or recording entry, and the retry exposes only the successful response.
  2. Stream a normal JSON object array and a numeric JSON array without protocol tags. Confirm that both are emitted and persisted unchanged, and that the numeric array starts streaming before the terminal event.
  3. Stream leading JSON before a real structured tool call, both with and without tool preparation metadata. Confirm the emitted and persisted order remains JSON text, then function call, with preparation metadata preserved. Also confirm that usage-only metadata passes through without blocking a later configured model fallback.

Evidence (Before & After)

Before: the production-shaped response was emitted as assistant text and persisted without a retry.

After: the same 905-character response produces PROTOCOL_TAG_LEAK, emits zero parts from the failed attempt, leaves history and recording unchanged, retries once, and exposes only the successful response.

Tested on

OS Status
🍏 macOS
🪟 Windows N/A
🐧 Linux N/A

Environment (optional)

Node.js 22, no sandbox. Verified with 321 affected unit tests, the full repository build and typecheck, ESLint, Prettier, and git diff --check.

Risk & Scope

  • Main risk or tradeoff: a response beginning with a JSON object or object array is buffered until it is disambiguated, so legitimate JSON with those shapes may be delivered at the terminal event rather than incrementally.
  • Not validated / out of scope: other malformed provider-specific tool syntaxes that do not match this production signature.
  • Breaking changes / migration notes: none.

Linked Issues

Fixes #8207

中文说明

本 PR 的改动

本 PR 防止模型将 JSON 数组形式的工具参数以及泄漏的 </parameter></function> 协议结束标签输出到 UI、会话历史或 session 记录中。失败轮次会复用现有的协议泄漏重试流程。

检测位于共享流处理边界,不再猜测参数名,而是暂存以 JSON 对象或对象数组开头的响应;在响应尚未判定时保持各 part 的顺序,并原样放行普通 JSON 或真实的结构化工具调用事件。数字数组和 Markdown 风格的方括号文本仍会立即流式输出。

为什么需要

一个生产会话中,模型以 finish_reason=stop 返回了包含两个 subagent 参数对象的纯文本 JSON 数组以及工具协议结束标签,但没有返回结构化工具调用。现有保护只处理以 XML 风格协议标签开头的响应,因此这个 JSON 变体被当作助手文本展示并持久化,没有触发重试。

Reviewer Test Plan

验证方式

  1. 流式返回 thought、分片 JSON 工具 payload 和 </parameter></function>,分别覆盖独立终止事件、finish reason 与内容同块、任意首个参数键、直接对象和尾随文本。确认失败轮次不输出任何 part、不写入历史或 recording,并且重试后只暴露成功响应。
  2. 流式返回不带协议标签的普通 JSON 对象数组和数字数组。确认两者原样输出和持久化,并且数字数组在终止事件前就开始输出。
  3. 在真实结构化工具调用前返回 JSON,分别覆盖带和不带工具 preparation 元数据的情况。确认输出与持久化顺序始终是 JSON 文本、function call,并保留 preparation 元数据;同时确认 usage-only 元数据会正常传递,且不会阻止后续配置的模型 fallback。

修复前后证据

修复前:生产形态的响应会直接作为助手文本输出并持久化,不触发重试。

修复后:相同的 905 字符响应触发 PROTOCOL_TAG_LEAK,失败轮次输出 0 个 part,history 和 recording 不变,随后重试且只暴露成功响应。

测试环境

OS Status
🍏 macOS
🪟 Windows N/A
🐧 Linux N/A

Node.js 22,无 sandbox。已通过 321 个受影响单元测试、全仓 build 与 typecheck、ESLint、Prettier 和 git diff --check

风险与范围

  • 主要风险或权衡:以 JSON 对象或对象数组开头的响应会暂存到完成判定,因此这些形态的合法 JSON 可能在终止事件时一次性输出,而不是增量输出。
  • 未验证或不在范围内:不符合本次生产特征的其他 provider 特有异常工具语法。
  • 破坏性变更或迁移说明:无。

关联 Issue

Fixes #8207

@qwen-code-ci-bot

qwen-code-ci-bot commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Qwen Triage finishedview run. See the stage comments in this thread for the result.

Qwen Triage 已完成 —— 查看运行。结果见本线程中的各阶段评论。

@github-actions github-actions Bot added the review/self-reported The linked issue was opened by the PR author (self-reported) label Aug 1, 2026
@wenshao

wenshao commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Code Review

Verified locally on PR head ea8984077 (fresh worktree): all 275 tests in packages/core/src/core/geminiChat.test.ts pass, including the 3 new cases. I also extracted LeadingProtocolTagLeakDetector verbatim into a standalone harness to probe its state machine, and ran one ad-hoc integration probe against sendMessageStream. Findings below reference those results.

Overview

Extends the existing LeadingProtocolTagLeakDetector from a 3-state to a 5-state machine (detecting | json | json-tool | clean | leaked) so that a leading JSON tool-argument array terminated by </parameter></function> is classified as a protocol leak and routed through the existing PROTOCOL_TAG_LEAK retry path instead of being emitted and persisted. Because the JSON case can only be disambiguated at the terminal event, the streaming loop gains a pendingProtocolParts queue that parks already-processed parts so ordering survives a late release. Good problem to fix — the reuse of the existing retry path is the right call, and the three new tests cover leak / ordinary-JSON / JSON-before-real-tool-call.


1. Detection is over-fitted to the single production sample (main concern)

JSON_TOOL_CALL_PREFIXES requires the payload to be an array whose first object's first key is one of four names. I ran the extracted detector over near-miss variants; the ones that slip through are not exotic:

Input Result
[{"name":…}]</parameter></function> ✅ LEAK
[{"prompt":…}]</parameter></function> ✅ LEAK
[\n {\n "name": …\n }\n]</parameter></function> (pretty-printed) ✅ LEAK
[{"file_path":"a.ts"}]</parameter></function> ❌ emitted as text
[{"command":"ls"}]</parameter></function> ❌ emitted as text
{"name":…}</parameter></function> (object, not array) ❌ emitted as text
[{"name":…}]</parameter></function>\nLet me continue. ❌ emitted as text
```json\n[{"name":…}]</parameter></function> ` ❌ emitted as text

Two of these look likely in practice: a different tool's first argument key (file_path, command, pattern, …), and trailing prose after the closing tags (the $ anchor in TOOL_CALL_CLOSING_TAGS requires the tags to be the very last thing in the buffer).

The </parameter></function> suffix is the unambiguous signal here; the opening key is not. Suggestion: enter the buffering state on any leading [ or {, and decide the leak in finish() purely on TOOL_CALL_CLOSING_TAGS — which lets JSON_TOOL_CALL_PREFIXES go away entirely and covers every key name. That does widen the class of responses whose streaming is deferred, so if you'd rather keep the buffering narrow, at minimum consider (a) matching the tool-arg keys anywhere in the first object rather than only as the first key, and (b) relaxing the $ anchor so trailing prose doesn't defeat detection.

2. takePendingProtocolParts(text) silently ignores text when the queue is non-empty

if (parts.length === 0) return text ? [{ text }] : [];
// …`text` is never used again

This is correct today only because of an unstated invariant: the concatenated text of pendingProtocolParts always equals the detector's released buffer. Every call site passes both, which reads as if they're combined. The invariant is easy to break — the GEMINI_EMPTY_CONTENT_PLACEHOLDER branch immediately above the loop is exactly the shape that would break it (an early continue that skips accept()), and it only stays safe because that part is excluded from both sides. Given how heavily commented the rest of this file is, this deserves either an explicit comment stating the invariant, a dev assertion, or a restructure where the detector buffer is the single source of truth and the queue carries only non-text parts.

3. Streaming stalls for the whole json-tool window

Once state === 'json-tool', accept() returns '' for every subsequent chunk until the terminal event — including thought parts, which get parked in pendingProtocolParts because the queue is non-empty. So a legitimate response that happens to start with [{"name": renders nothing (not even reasoning) until the stream finishes. The PR description calls this out, but note the class is a bit wider than "narrow": acceptJsonCandidate lowercases and strips all whitespace before matching, so [{"Name":, [ { "name" : , and pretty-printed variants are all captured. There's also no upper bound on buffer growth. A max-buffer guard that force-releases into clean would bound the worst case (memory is bounded by max output tokens, so this is a UX rather than a safety issue).

Positive note: accept() returns early in json-tool state before recomputing trimStart().toLowerCase(), so there's no O(n²) rescan of the buffer. Worth keeping that early return where it is.

4. Verified: a usage-only chunk arriving mid-buffer is dropped from the yielded stream

The gate is coarse:

if (!protocolTextWasSuppressed || !protocolTagDetector.blockingOutput) yield chunk;

blockingOutput is state !== 'clean', which now stays true until the terminal event. I confirmed with an integration probe ([{"name":…}] chunk → usage-only chunk → stop chunk): 0 usage-bearing chunks reach the consumer, versus 1 on main. Impact is limited — usageMetadata is captured into the local at the top of the loop before the gate, and turn.ts reads usage off the finish-reason chunk — so nothing user-visible today. But the gate is now suppressing chunks it has no reason to inspect. Consider skipping the gate for chunks with no candidates.

5. Undocumented behavior change on the pre-existing XML leak path

Previously if (typeof part.text !== 'string' || part.thought) return [part]; let thought and non-text parts through even after leaked. Now they're pushed onto pendingProtocolParts and discarded when the leak is confirmed. That's consistent with "the failed attempt emits nothing" and is probably an improvement, but it changes the <analysis>/<summary> path too, which is outside the PR's stated scope. Worth an explicit test pinning it so a future refactor doesn't silently flip it back.

6. Minor

  • finish() is called twice on one path. When the terminal chunk has content but no parts, the new branch calls finish(), reassigns content with the released parts, and then if (content?.parts) re-enters the loop and calls finish() again. Harmless (the second call short-circuits on state !== 'detecting'), but confusing to read — a comment would help.
  • Re-feeding released parts through accept(). Same path: the parts that finish() just released are pushed back through protocolTagDetector.accept(part.text). Safe only because release() already set state = 'clean'. Another implicit invariant.
  • pendingProtocolParts.push(...outputParts.splice(0), part) — moving already-accumulated output back into the pending queue is the least obvious line in the diff and has no comment.
  • Naming. TOOL_CALL_CLOSING_TAGS matches an anchored suffix, not "closing tags" generally; LEAKED_TOOL_CALL_SUFFIX_RE (or similar) would signal the $ anchor. releaseJsonCandidate() is public but undocumented — a one-line doc ("flush the ambiguous JSON buffer because a non-text part proved this is a real response") would help.
  • The flatMap → imperative-loop rewrite makes the diff larger than the behavior change; that's justified here by the early-continue needs, just noting it for reviewers scanning the diff size.

Test coverage

The three new tests are well-targeted and the it.each parameterization over expectedTextChunks neatly documents the latency tradeoff. Gaps worth closing:

  • Leak text and finishReason in the same chunk. Both new leak tests use a separate terminal {finishReason: 'STOP'} event. The other path (content.parts present and finishReason set, hitting finish() at the end of the loop) isn't covered, and some providers emit it that way.
  • Negative tests pinning the scope. [{"file_path":…}]</parameter></function>, the single-object form, and trailing-prose-after-tags all currently pass through. Whether that's intended or a follow-up, a test would document it.
  • Stream ends while buffering with no finishReason. finish() never runs, pendingProtocolParts is silently dropped, and the turn fails with NO_FINISH_REASON instead. Reasonable, but untested.
  • LeadingProtocolTagLeakDetector isn't exported, so every state-machine case has to go through the full sendMessageStream mock harness (~2s each because of the retry delay). Exporting it for direct unit tests would make covering the matrix in section 1 nearly free.

Security

No new attack surface. The change is strictly suppress-and-retry; the risk direction is false negatives (leaked protocol text reaching history), not false positives leaking anything out.


Verdict: the approach is sound and the retry-path reuse is right. I'd want section 1 addressed before merge (the file_path / trailing-prose misses will show up as the same bug report again), and section 2 documented. Everything else is polish.

中文摘要

已在 PR head ea8984077 上本地验证geminiChat.test.ts 全部 275 个测试通过(含 3 个新增用例);另将检测器单独抽出做了状态机探测,并跑了一个针对 sendMessageStream 的集成探针。

  1. 检测规则过度贴合单个生产样本(主要问题)。只匹配「数组 + 首个对象的首个 key ∈ 4 项白名单」。实测漏检:[{"file_path":…}][{"command":…}]、单对象 {"name":…}、结束标签后还有正文、以及 ```json 代码围栏。</parameter></function> 才是可靠信号,建议以它为判据(首字符为 `[`/`{` 即进入缓冲,`finish()` 时按结束标签判定),可直接删掉 `JSON_TOOL_CALL_PREFIXES`。至少也应放宽 `$` 锚点并扩大 key 匹配范围。
  2. takePendingProtocolParts(text) 在队列非空时会静默丢弃 text 参数。目前正确依赖一个未写明的不变式(pending 各 part 文本拼接 == 检测器缓冲区),很脆弱,建议加注释或断言。
  3. json-tool 状态下整段流式输出停滞(含 thought),且缓冲无上限;由于会先 lowercase + 去除全部空白,受影响的合法 JSON 比描述中更广。
  4. 已验证:缓冲期间到达的 usage-only chunk 不会被 yield(探针实测 0 个,main 上为 1 个)。当前无用户可见影响,但建议对无 candidates 的 chunk 跳过该 gate。
  5. 顺带改变了原有 XML 泄漏路径的行为:leaked 后的 thought / 非文本 part 现在会被丢弃(此前会透传)。方向上是改进,但超出本 PR 声明范围,建议补测试固化。
  6. 次要:终止 chunk 上 finish() 被调用两次;已释放的 part 会再次过一遍 accept()pendingProtocolParts.push(...outputParts.splice(0), part) 缺注释;TOOL_CALL_CLOSING_TAGS 命名未体现 $ 锚点。

测试建议补充:泄漏文本与 finishReason 同 chunk 的情形;上述漏检变体的负向用例(固化范围);缓冲中流意外结束(无 finishReason);导出 LeadingProtocolTagLeakDetector 以便直接做状态机单测。

结论:思路正确、复用现有重试路径合理。建议合并前处理第 1 点、补充第 2 点的说明,其余为打磨项。

@qwen-code-ci-bot

qwen-code-ci-bot commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the PR!

Template looks good ✓

Problem: observed production bug. Issue #8207 carries the actual leaked payload (subagent dispatch JSON rendered as assistant text), the version (0.21.0-preview.2), the model (qwen3.7-max), and the triggering conditions (6th consecutive tool-call turn, ~35K input tokens, a 429 retry across pool nodes). A maintainer confirmed the gap in source. This is not theoretical.

Direction: aligned. This extends the existing LeadingProtocolTagLeakDetector / PROTOCOL_TAG_LEAK retry architecture with a JSON-shaped variant — same class of model format degradation, same recovery path. The issue's coverage table shows no existing guard matches this variant.

Size: 187 production lines (geminiChat.ts: 157+/30−), 689 test lines (geminiChat.test.ts: 685+/4−). Well under the 500-line threshold — no maintainer escalation needed.

Approach: the scope feels right. The detector gains a 'json' buffering state, a hasLeakedToolCallTags() scanner that tracks JSON string boundaries to avoid false positives, and pendingProtocolParts to preserve part ordering during ambiguity. Every edit serves the stated goal — no drive-by refactors or unrelated changes. The isToolCallPreparationOnlyhasCandidateOutput rename is a small behavior correction (usage-only metadata no longer counts as a "yielded chunk") that the buffering logic depends on, so it belongs here.

Risk: geminiChat.ts matches the high-risk path list (10 of 31 reverted PRs touched these paths). This doesn't block, but it means the code review and CI evidence need to be solid — see Stage 2.

Moving on to code review. 🔍

中文说明

感谢贡献!

模板完整 ✓

问题:已观测到的生产 bug。Issue #8207 附带了真实泄漏的 payload(subagent 调度 JSON 被当作助手文本渲染)、版本(0.21.0-preview.2)、模型(qwen3.7-max)以及触发条件(第 6 轮连续 tool call、约 35K 输入 token、跨池节点 429 重试)。维护者已在源码中确认缺口。非理论性问题。

方向:对齐。在现有 LeadingProtocolTagLeakDetector / PROTOCOL_TAG_LEAK 重试架构上扩展 JSON 变体——同类模型格式退化,同一恢复路径。issue 中的覆盖表显示现有保护均不匹配此变体。

规模:187 行生产代码(geminiChat.ts: 157+/30−),689 行测试(geminiChat.test.ts: 685+/4−)。远低于 500 行阈值,无需维护者升级。

方案:范围合理。检测器新增 'json' 缓冲状态、hasLeakedToolCallTags() 扫描器(跟踪 JSON 字符串边界以避免误判)、以及 pendingProtocolParts 保持歧义期间的 part 顺序。所有改动服务于既定目标——无顺手重构或无关变更。isToolCallPreparationOnlyhasCandidateOutput 重命名是缓冲逻辑所依赖的小型行为修正(usage-only 元数据不再计为"已输出块"),属于本 PR 范围。

风险:geminiChat.ts 命中高风险路径列表(31 个被 revert 的 PR 中有 10 个触及这些路径)。不阻塞,但代码审查和 CI 证据需要扎实——见 Stage 2。

进入代码审查 🔍

Qwen Code · qwen3.8-max-preview

Reviewed at 45ed8f111ad5868c2903e2b1f9ac9a7a40d142b2 · re-run with @qwen-code /triage

@qwen-code-ci-bot

qwen-code-ci-bot commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Code Review

Independent proposal. Given the problem — JSON-serialized tool arguments leaking as plain text when the model drops function-calling format — I would extend the existing LeadingProtocolTagLeakDetector with a JSON buffering state: buffer leading { or [{ content until the stream finishes, then check for `

@qwen-code-ci-bot

qwen-code-ci-bot commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Confidence: 4/5 — solid, well-tested fix for a real production bug; the implementation is the minimal change the problem needs, CI is green, and the only reservation is that geminiChat.ts is a historically revert-prone file where a behavioural claim deserves sandboxed verification.

Stepping back: the problem is real (production payload in #8207, maintainer-confirmed gap), the approach matches my independent proposal almost exactly, and the test suite is thorough enough that I can trace every code path through a test. The hasLeakedToolCallTags string-boundary tracking is the kind of detail that separates a fix from a source of new false positives, and it's tested. The !hasToolCall guard on the final leak throw is a subtle correctness point — without it, a stream that emits a real tool call followed by leaked text would retry and lose the tool call — and there's a dedicated test for it.

The isToolCallPreparationOnlyhasCandidateOutput rename is the only change that isn't strictly about JSON leak detection, but the buffering logic depends on it (usage-only chunks must pass through without being swallowed by the protocol detector), and the updated test pins the new behavior. It's a small, correct behavior change that belongs here.

If I had to maintain this in six months, the state machine is clear enough: detecting → json → clean/leaked, with release() as the single clean-exit path. The pendingProtocolParts buffering adds a second layer of state, but it's scoped to the stream loop and doesn't leak into the detector itself.

Non-blocking note: the buffering tradeoff (leading JSON objects/arrays delivered at the terminal event rather than incrementally) is documented in the PR description and is the right call — correctness over latency for an ambiguous prefix.

Sandboxed verification would settle the remaining behavioural gap: @qwen-code /verify — that the 905-character production-shaped response from #8207 actually produces PROTOCOL_TAG_LEAK and retries cleanly is substantiated by unit tests but not by an A/B load-bearing proof against the base build. This is a sponsored run (fork PR — a maintainer triggers it); read the resulting report with the same skepticism as the fork's own CI logs.

中文说明

置信度:4/5 —— 针对真实生产 bug 的扎实、充分测试的修复;实现是问题所需的最小变更,CI 绿色,唯一的保留是 geminiChat.ts 是历史上容易被 revert 的文件,行为性声明值得沙箱验证。

回顾全局:问题是真实的(#8207 中的生产 payload,维护者确认的缺口),方案与我的独立提议几乎完全一致,测试套件足够详尽,每条代码路径都能追溯到测试。hasLeakedToolCallTags 的字符串边界跟踪是区分"修复"和"新的误判来源"的关键细节,且有测试覆盖。最终泄漏抛出上的 !hasToolCall 保护是一个微妙的正确性要点——没有它,一个先发出真实工具调用再跟随泄漏文本的流会重试并丢失工具调用——有专门的测试覆盖。

isToolCallPreparationOnlyhasCandidateOutput 重命名是唯一不严格属于 JSON 泄漏检测的变更,但缓冲逻辑依赖它(usage-only 块必须透传而不被协议检测器吞掉),更新后的测试固定了新行为。这是一个小型、正确的行为变更,属于本 PR。

如果六个月后维护这段代码,状态机足够清晰:detecting → json → clean/leakedrelease() 作为唯一的干净退出路径。pendingProtocolParts 缓冲增加了第二层状态,但作用域限于流循环,不会泄漏到检测器本身。

非阻塞说明:缓冲权衡(前导 JSON 对象/数组在终止事件时一次性交付而非增量交付)已在 PR 描述中记录,是正确的选择——对歧义前缀,正确性优先于延迟。

沙箱验证可以弥补剩余的行为缺口:@qwen-code /verify —— #8207 中 905 字符的生产形态响应是否确实触发 PROTOCOL_TAG_LEAK 并干净重试,已由单测证实,但缺少对 base build 的 A/B 承重证明。这是赞助运行(fork PR——由维护者触发);请以与 fork CI 日志相同的审慎态度阅读生成的报告。

Qwen Code · qwen3.8-max-preview

Reviewed at 45ed8f111ad5868c2903e2b1f9ac9a7a40d142b2 · re-run with @qwen-code /triage

@yiliang114

Copy link
Copy Markdown
Collaborator Author

Addressed in f3fd84a. Detection now buffers a leading JSON object or object array independently of its first key, while numeric arrays and Markdown-style brackets release immediately; the closing protocol tags are also recognized before optional trailing prose. I removed the unused released-text parameter from pending-part replay, preserved usage-only chunks without counting them as user-visible fallback progress, and added regressions for file_path-first arrays, direct objects with trailing prose, both finish boundaries, ordinary JSON streaming, structured-call/preparation ordering, and usage-only fallback behavior. The focused suites pass 318/318, and the exact 905-character production payload from the trace is fully suppressed and retried.

@wenshao

wenshao commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Code Review (round 2)

Verified locally on PR head f3fd84a04 in a detached worktree: packages/core/src/core/geminiChat.test.ts passes 276/276, and the whole packages/core/src/core suite passes 2640/2640 (57 files). I also re-extracted LeadingProtocolTagLeakDetector into a standalone harness for state-machine probing, and ran three ad-hoc integration probes through sendMessageStream (added to a copy of the test file, not to the branch). Every finding below is from those runs, not from reading alone.

Previous round — status

Prior finding Status at f3fd84a04
1. Detection over-fitted to one production sample (JSON_TOOL_CALL_PREFIXES, $ anchor) Fixed — key whitelist gone; [{"file_path":…}], direct objects, and trailing prose after a newline are all caught now
2. takePendingProtocolParts(text) silently ignoring text Fixed — parameter removed
4. usage-only chunk dropped from the yielded stream while buffering Fixed — `!chunk.candidates?.length
3. Streaming stalls for the buffering window Still open, and now wider — see §2
5. Leaked-path thought/non-text parts now discarded (XML path too) Still unpinned by a test
6. finish() called twice; released parts re-fed through accept(); outputParts.splice(0) uncommented Still open (all cosmetic)

Also worth calling out as an unadvertised improvement: the new candidate?.finishReason && !content?.parts branch means finish() now runs when the terminal event arrives as a separate {finishReason}-only chunk. Previously finish() was only reachable from inside if (content?.parts), so a buffered leading tag at that boundary was dropped and surfaced as NO_RESPONSE_TEXT rather than PROTOCOL_TAG_LEAK. Good fix — but it's behavior change in the XML path that no test pins.


1. Critical — the leak scan runs over the whole buffered response, so a legitimate JSON answer that merely mentions the tags is discarded and retried

Removing the key whitelist widened the buffering trigger to any leading { or [{. finish() then tests LEAKED_TOOL_CALL_TAGS against the entire buffer — anywhere in it, not just where the leading JSON value ends. So a valid JSON response whose body contains } + </parameter> + </function> in a string is classified as a leak.

Verified end-to-end through sendMessageStream (2 calls to the content generator, original response never reaches the consumer):

PROBE-FP  {"verdict":"fail","evidence":"the model emitted } </parameter> </function> mid-turn"}
          → retried=true  calls=2  lastText="retried"
PROBE-FP  {\n  "example": "a call ends with }</parameter></function> then stops"\n}
          → retried=true  calls=2  lastText="retried"

This is reachable, not theoretical. forkedAgent sets responseMimeType: 'application/json' + responseJsonSchema and runs the query through GeminiChat.sendMessageStream, so every structured-output subagent/judge response begins with { and enters the buffering state. Any such response that quotes or describes a leaked tool-call payload — for example a verifier reporting on the very failure mode this PR is about — gets silently thrown away and retried up to protocolTagLeakMaxRetries. Because the model will regenerate the same content, the retries burn and the turn then fails with the leak error.

Suggested fixes, in order of preference:

  1. Anchor the decision to the end of the leading JSON value: scan the buffer for the balanced end of the first {…} / […] (or JSON.parse the prefix), then only accept the tags immediately after it. This keeps every true positive in the PR's own tests and removes the whole class of false positives.
  2. Cheaper stopgap: don't buffer at all when the request set responseJsonSchema / responseMimeType: 'application/json' — a structured-output turn can never be a tool-protocol leak.

Either way, the mid-buffer match should not be enough on its own.

2. Suggestion — the buffering window is now the whole response for every {-leading answer

accept() enters json on a bare leading { and from then on returns '' for every chunk until the terminal event, and non-text/thought parts land in pendingProtocolParts once the queue is non-empty. So a model asked to emit a JSON config or a structured-output subagent renders nothing — not even reasoning — until the stream finishes. The PR body describes this, but the class is materially wider than in the previous round (previously gated by a 4-key whitelist and an array-only prefix).

There is still no upper bound on buffer growth. Bounded by max output tokens, so it's a UX rather than a safety issue, but a max-buffer force-release into clean would cap the worst case. Fixing §1 by anchoring to the end of the leading JSON value would also let you release the buffer as soon as that value closes, which removes most of this stall.

Positive: the if (this.state === 'json') return ''; early return still sits before the trimStart().toLowerCase() recompute, so there is no O(n²) rescan. Keep it there.

3. Suggestion — residual false negatives in LEAKED_TOOL_CALL_TAGS

Verified with the extracted detector and one integration probe:

Input Result
[{…}]\n</parameter>\n</function>\n (production shape) LEAK ✅
[ { … } ]\n</parameter>\n</function>\n (pretty-printed) LEAK ✅
[{…}]</parameter></function>Let me continue. emitted as text
{"name":"x"</parameter></function> (truncated JSON, no closing brace) emitted as text
```json\n[{…}]\n</parameter></function> emitted as text (out of scope, fine)
PROBE-MISS retried=false lastText="[{\"name\":\"x\"}]</parameter></function>Let me continue."

The (?:\s|$) tail is doing no disambiguation work — </function> already ends in >. Dropping it costs nothing and closes the first miss. The [}\]] prefix requirement is what loses the truncated-payload case; if you keep it, that's a deliberate narrowing worth a negative test so it doesn't read as an oversight.

4. Suggestion — hasCandidateOutput changed more than the rename suggests

isToolCallPreparationOnly returned true only when preparations existed and there was no candidate output and no usageMetadata. hasCandidateOutput drops both the preparations precondition and the usageMetadata term, and both call sites use it inverted. Net effect beyond the intended usage-metadata fix: a chunk with no preparations, no parts and no finish reason used to count as "the stream yielded something" and now does not, at both the main-send site and the fallback site. That makes empty-stream detection stricter (more retries/fallbacks) — probably the behavior you want, but it is a wider change than "usage-only metadata passes through", and only the usage-only half is covered by the renamed test. Worth a line in the PR body and a test for the no-preparations/no-parts chunk. I grepped packages/*/src for other isToolCallPreparationOnly readers — only the two migrated sites (remaining hits are dist/ and coverage artifacts).

5. Minor

  • The detector buffer and pendingProtocolParts are two copies of the same text kept in sync implicitly. finish()'s and releaseJsonCandidate()'s return values are now used purely as booleans (protocolTagDetector.finish(); with the result discarded at two sites), and the actual text comes from the parked parts. This is correct today only because every accept() that returns '' is paired with a pendingProtocolParts.push(part). The GEMINI_EMPTY_CONTENT_PLACEHOLDER continue right above the loop is exactly the shape that breaks that pairing, and it is safe only because it is excluded from both sides. Given how heavily commented the rest of this file is, this deserves an explicit invariant comment or a restructure where the buffer is the single source of truth.
  • finish() is still called twice on the terminal-chunk-without-parts path (once in the new branch, once at the end of the parts loop after the released parts are re-entered). Harmless — the second call short-circuits — but a one-line comment would save the next reader the trace.
  • pendingProtocolParts.push(...outputParts.splice(0), part) remains the least obvious line in the diff and is still uncommented. Note it also parks an already-approved functionCall from earlier in the same chunk, delaying dispatch to the terminal event.
  • releaseJsonCandidate() is public and still undocumented; one line ("a non-text part proved this is a real response — flush the ambiguous JSON buffer") would carry it.
  • Naming: LEAKED_TOOL_CALL_TAGS is a big improvement over TOOL_CALL_CLOSING_TAGS.

Test coverage

Good progress: leak-text-and-finishReason-in-the-same-chunk is now covered via finishWithContent, the ordering test asserts usage metadata passthrough, and the expectedTextChunks parameterization documents the latency tradeoff nicely. Remaining gaps:

  • A legitimate leading {…} object with no tags. The ordinary-JSON test only covers arrays, yet { is the trigger that now buffers unconditionally. One case pinning "object streams through, released at finish" is cheap.
  • The false positives in §1 — whichever way you resolve them, they belong in the suite.
  • Negative test for §3's misses, to pin the intended scope.
  • Stream ends while buffering with no finishReason. finish() never runs, pendingProtocolParts is silently dropped, and the turn fails as NO_FINISH_REASON. Reasonable, still untested.
  • LeadingProtocolTagLeakDetector is still not exported. Every state-machine case costs a full sendMessageStream mock round (~2s each with the retry delay). Exporting it would make the matrices in §1 and §3 nearly free — that is what made the probing above practical for me.

Security

No new attack surface; the change is suppress-and-retry. §1 does shift the risk direction, though: with the whitelist removed the failure mode is no longer only "a leak reaches history" but also "a valid response is destroyed", which is the more user-visible of the two.


Verdict: the round-1 findings were addressed well and the direction is right. §1 is the one I'd want fixed before merge — it is verified, reachable through the structured-output path, and it trades a false negative for a false positive that silently deletes correct model output. §3 is a two-character fix. Everything else is polish or tests.

中文摘要

已在 PR head f3fd84a04 本地验证geminiChat.test.ts 276/276 通过,packages/core/src/core 全量 2640/2640 通过;另将检测器抽出做状态机探测,并通过 sendMessageStream 跑了 3 个集成探针。

上一轮问题:第 1(过度贴合单样本)、第 2(text 参数被忽略)、第 4(usage-only chunk 被吞)已修复;第 3、5、6 仍在。新增亮点:终止 chunk 无 parts 时现在也会调用 finish(),补上了原有的一个漏洞。

  1. Critical:泄漏判定扫描整个缓冲区,导致合法 JSON 响应被误判丢弃并重试。 去掉 key 白名单后,任何以 { / [{ 开头的响应都会整体缓冲,finish()全缓冲区任意位置匹配 }</parameter></function> 即判为泄漏。实测(探针):{"verdict":"fail","evidence":"… } </parameter> </function> …"}retried=true, calls=2,原始响应完全丢失。该路径真实可达:forkedAgent 的结构化输出(responseMimeType: 'application/json')走的就是 GeminiChat.sendMessageStream,响应必然以 { 开头;若内容里引用了协议标签(例如判定/复述本 issue 的场景),会被反复重试直至耗尽。建议:先定位首个 JSON 值的闭合位置(括号配平或对前缀 JSON.parse),只接受紧随其后的标签;或至少在请求带 responseJsonSchema 时完全跳过缓冲。
  2. 缓冲窗口现在覆盖所有以 { 开头的响应(含 thought 全部滞留),且缓冲无上限。修好第 1 点(值闭合即释放)可顺带缓解。
  3. 仍存在漏检</function> 后紧跟正文(无空白)不匹配;缺少闭合括号的截断 payload 不匹配。(?:\s|$) 尾部约束没有区分作用,可直接去掉。
  4. hasCandidateOutput 的语义变化大于重命名:对于「无 preparation、无 parts、无 finishReason」的 chunk,判定从 true 翻转为 false,两个调用点都受影响。方向合理,但超出 PR 描述,建议补测试与说明。
  5. 次要:检测器 buffer 与 pendingProtocolParts 是隐式同步的两份同源数据(finish() 返回值已退化为布尔),建议写明不变式;finish() 仍被调用两次;push(...outputParts.splice(0), part) 仍无注释;releaseJsonCandidate() 缺文档。
  6. 测试建议:补合法单对象 JSON 直通用例、第 1/3 点的正反用例、缓冲中无 finishReason 结束的用例;并导出 LeadingProtocolTagLeakDetector 以便直接做状态机单测。

结论:整体方向正确,上一轮问题处理得好。建议合并前修复第 1 点(已实测、可达、会静默删除正确输出),第 3 点改动极小,其余为打磨与测试。

@yiliang114

Copy link
Copy Markdown
Collaborator Author

Addressed the round-2 findings in ca111fc8c47.

  • Closing-tag detection is now quote/escape aware, so literal </parameter></function> examples inside JSON strings are preserved.
  • releaseJsonCandidate() reuses the terminal leak check, closing the cross-chunk JSON → structured-call bypass.
  • Once a structured tool call has already been emitted, a later leaked text suffix is suppressed without retrying and risking duplicate side effects.
  • Closing tags followed immediately by prose are detected without requiring whitespace.

The new regressions were exercised red-first. I also reran the exact 905-byte production response across five chunkings (single chunk, after byte 1, after byte 2, before </parameter>, and character-by-character): the failed attempt reached neither streamed output, history, nor recording, and only the successful retry remained. Final focused suites pass 321/321, with repository build, typecheck, ESLint, Prettier, and git diff --check green.

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no blockers. Suggestions are inline.

中文说明

已审查——无阻断问题。 建议见行内评论。

— qwen3.8-max-preview via Qwen Code /review

Comment on lines +1192 to +1194
if (escaped) escaped = false;
else if (char === '\\') escaped = true;
else if (char === '"') inString = false;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] The escape-handling branch (escaped flag) in hasLeakedToolCallTags has no test coverage. The only string-interior test uses a value with no backslash escapes. — Failure scenario: deleting the two escape lines as a mutation causes a false positive on input like {"example":"\"}</parameter></function>"} — the mutant treats \" as the string’s closing quote, sees the following } outside a string, matches the leak regex, and triggers an incorrect retry that discards a valid model response. The present code is correct; this is a coverage gap with a concrete surviving mutant.

Suggested test addition:

it('does not reject closing protocol tags after an escaped quote inside a JSON string', () => {
  const text = JSON.stringify({ example: '\\"}</parameter></function>' });
  // assert no retry, text preserved
});
中文说明

hasLeakedToolCallTags 中的转义处理分支(escaped 标志)没有测试覆盖。唯一的字符串内部测试使用的值不包含反斜杠转义。删除这两行转义代码作为变异,会导致对包含转义引号的输入产生误报,触发不必要的重试并丢弃有效的模型响应。当前代码是正确的,这是一个有具体存活变异的覆盖缺口。

— qwen3.8-max-preview via Qwen Code /review

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid coverage gap; the current implementation is correct, and the final review found no blocker. This PR has already gone through more than five review/fix rounds with the behavioral paths and CI green, so I am deferring this non-blocking mutation-only test instead of resetting the full review and CI loop again.

@wenshao

wenshao commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Review: fix(core): retry leaked JSON tool protocol output

Overview

Extends LeadingProtocolTagLeakDetector with a json state: a response whose first non-whitespace text is { or [{ is buffered instead of streamed, and at the terminal event the buffer is scanned (string-aware) for a closing brace/bracket immediately followed by </parameter></function>. A match marks the turn leaked and reuses the existing PROTOCOL_TAG_LEAK retry path; otherwise the buffer is released unchanged. A new pendingProtocolParts queue preserves part ordering across the buffering window, and isToolCallPreparationOnly is replaced by the simpler hasCandidateOutput.

The approach is sound — detecting at the shared stream boundary, not guessing argument names, and reusing the existing retry path are all the right calls. Findings below, roughly by severity.


1. releaseJsonCandidate() finalizes on a partial buffer — the leak escapes

releaseJsonCandidate() calls finish(), which runs hasLeakedToolCallTags against whatever is in the buffer right now and then transitions to clean permanently. It is invoked from two places that can fire mid-payload:

  • the tool-call-preparation branch (geminiChat.ts ~L4162), and
  • any non-text part inside content.parts (~L4210).

If either fires before the closing tags have streamed in, the partial JSON is released as clean and everything after it — including </parameter></function> — streams straight to the UI and history. Confirmed with the detector in isolation:

premature release mid-JSON    leaked=false
streamed="[{\"name\":\"a\"}]\n</parameter>\n</function>"

(12 chars accepted → releaseJsonCandidate() → remainder accepted in clean state.)

This is reachable exactly in the shape the PR targets: a provider that emits tool-call preparation metadata and then abandons the structured call, emitting the payload as text. Suggestion: only finalize when the buffer is terminal — either re-enter json state if more text arrives after a speculative release, or have releaseJsonCandidate() release without permanently clearing the json state so the tag check still runs at finish().

2. Detection window is narrower than the PR body implies

LEAKED_TOOL_CALL_TAGS = /^[}\]]\s*<\/parameter>\s*<\/function>/i requires the tags to sit immediately after a closing brace/bracket (whitespace only). Anything in between escapes. Probes:

input detected
[{…}]\n</parameter>\n</function>
{…}\n\n</parameter></function>
{"a":1}\nDone.\n</parameter>\n</function>
```json\n{…}\n``` \n</parameter></function> ❌ (never enters json state)

The false-positive side is correspondingly tight — {"a":1} the model emitted </parameter></function> here is not flagged, and the string-aware scan handles tags inside JSON strings (good, and tested). So the tradeoff is deliberate; it's just worth stating in the code that this matches one production signature rather than a class of them, since a false positive burns protocolTagLeakMaxRetries and then hard-fails the turn.

3. Buffering surface grew a lot, with no cap and no non-terminal flush

Previously only text starting with <analysis / <summary was withheld. Now every response starting with { or [{ is buffered in full until the terminal event, and takePendingProtocolParts also retracts same-chunk thought parts (pendingProtocolParts.push(...outputParts.splice(0), part), ~L4232) — so thinking text stops streaming live for those responses too. Consequences worth considering:

  • Unbounded buffer growth on a large JSON answer, delivered as one part at the end. A size cap that releases as clean past some threshold would be a cheap safety valve.
  • The buffer is only flushed on a finishReason chunk. If the stream ends without one, the buffered text never reaches allModelParts. That's usually covered by the NO_FINISH_REASON throw — except when hasToolCall is true, where validation passes and the buffered text is silently dropped from both the emitted stream and history.
  • On mid-stream streamError, the buffered text is likewise absent from the partial assistant turn used by the repair path.

None of these is covered by a test.

4. !hasToolCall on the throw is a good fix, but the leak is now silent

Adding && !hasToolCall (~L4471) correctly stops a retry after a tool call has already been yielded downstream — that would otherwise re-execute the call. Nice catch, and the new test documents it.

The side effect: in that ordering the leaked text is swallowed (pendingProtocolParts = []) and nothing is logged. Every other rejection path in this file emits a debugLogger.warn. A one-line warn here would make the "text vanished but no retry" case diagnosable in the field. Also note the behavior is now order-dependent (leak→tool-call retries, tool-call→leak does not); a comment stating that intent would help the next reader.

5. isToolCallPreparationOnlyhasCandidateOutput is a wider change than the rename suggests

The predicate isn't the inverse of the old one. Two new cases now count as "no user-visible output":

  • usage-only chunks (no candidates) — this is the intended fix, and it's fine: turn.ts only surfaces usageMetadata on the Finished event, which needs a finishReason, so nothing user-visible is lost.
  • chunks with candidates whose parts is [] and no finishReason, regardless of whether preparations are present — previously these counted as output.

Both make the fallback chain more likely to run. The knock-on I'd double-check is currentFallbackYieldedAnyChunk (~L3401) gating popPendingPartialAssistantTurn() in both directions (L3419 and L3468) — the reasoning that a usage-only chunk can never coexist with a pushed partial turn holds as far as I can tell, but it's load-bearing and undocumented.

6. Nits

  • Aliasing: at ~L4226 the released text part is pushed by reference (outputParts.push(...takePendingProtocolParts(), part)), whereas every other release path copies ({ ...part, text }, and takePendingProtocolParts itself does { ...part }). Since the history consolidation later mutates in place (lastPart.text += part.text), a raw reference that was already yielded can have its text mutated after the fact. { ...part } here would keep the invariant uniform.
  • hasLeakedToolCallTags has no comment explaining why it tracks string/escape state — that's the whole point of the function and the reason the } </parameter></function>-inside-a-string test passes. Two lines would help.
  • The class name LeadingProtocolTagLeakDetector and the constant LEAKED_TOOL_CALL_TAGS no longer describe what they do (the latter is really "closing-delimiter-adjacent protocol tags"); PROTOCOL_TAG_PREFIXES sits right above and now covers only half the detector's job.
  • Perf is a non-issue — I benchmarked hasLeakedToolCallTags at ~4 ms on a 768 KB payload despite the per-} slice() (V8 sliced strings), so no change needed.

Test coverage

Good breadth on the happy paths: leaked array/object with and without a terminal event, ordinary object array and numeric array (with an explicit assertion that the numeric array streams incrementally — nice), ordering vs. a real structured tool call with and without preparation metadata, tags inside a JSON string, and both tool-call orderings. Converting streamResponse to varargs is a clean way to get there.

Missing, mapping to the findings above:

  • a preparation chunk or non-text part arriving mid-payload (finding 1)
  • a buffered JSON response whose stream ends without a finishReason, and one that errors mid-stream (finding 3)
  • finishWithContent: true is only exercised for the single-object case; the array case only covers a separate terminal event

Security / correctness posture

No injection or data-exfiltration surface; the change is defensive and fails toward retry. The main correctness risk is silent data loss (findings 3 and 4) rather than incorrect output.


Overall: the direction is right and the ordering machinery is carefully done. Finding 1 is the one I'd want addressed before merge — it's a hole in the exact path the PR is built to close.

@yiliang114

Copy link
Copy Markdown
Collaborator Author

Real session JSONL verification

I pulled the complete production session artifact from the trace-derived Beijing OSS bucket and replayed it directly through sendMessageStream.

Artifact checks:

  • 144,472 bytes, 26/26 lines parsed as JSON
  • SHA-256: fb0f25c92713e7c0fc3c240f24a7469c3cb649a8f0269041d25802186e5471ca
  • The final persisted assistant text is 904 bytes and exactly matches the 905-byte SLS model response after removing its trailing newline
  • The payload is the reported JSON-array tool protocol followed by the closing parameter / function tags

Before the fix (a4c0632c420), all five replay boundaries made only one upstream call, emitted no retry event, and exposed the failed payload in both the live stream and chat history. The five boundaries were: one chunk, split after character 1, split after character 2, split immediately before the closing protocol tags, and one character per chunk.

At the current head (ca111fc8c4771c237d659bf4a3c1e4829dfc1675), I ran the same five JSONL-backed cases twice (10/10 passed). Every case made two upstream calls, emitted the retry event, and kept the failed payload out of the live stream, chat history, and assistant-turn recording; only the successful retry response remained.

This is in addition to the existing 321/321 affected tests, build, typecheck, formatting/lint checks, and the green GitHub CI/review runs at the same head.

@wenshao

wenshao commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Local real-stack verification (merge reference)

Full end-to-end run on macOS (Node v24), no mocks inside the product: two fresh detached worktrees — before = PR base a4c0632c420, after = PR head ca111fc8c47 — each npm ci + full npm run bundle, launched as the real interactive TUI (node dist/cli.js inside tmux 120×40, isolated $HOME, auth via OPENAI_* env). Upstream is a local OpenAI-compatible SSE server that replays the #8207 production signature: reasoning_content thought → the two-subagent JSON array split across 4 content chunks → \n</parameter>\n</function>\nfinish_reason=stop + usage, and serves a clean response to any retry. Every upstream request is logged to JSONL; session persistence is checked in ~/.qwen/projects/*/chats/*.jsonl. All screenshots, the mock server, the runner script, and the raw request logs are on pr-assets/8301-verify for replay.

1. Before (base a4c0632c420) — leak reproduced

One upstream call, no retry. The JSON payload and the closing protocol tags render as assistant text, and the session JSONL persists them (2 × </parameter> in the chat file).

before: leaked JSON payload and protocol tags rendered as assistant text

2. After (PR head ca111fc8c47) — fixed for the #8207 signature

Request log shows the retry doing its job — same conversation sent twice, 2.3 s apart (leak attempt, then retry):

{"n":1,"t":"…15:13:09.209Z","mode":"leak","trigger":true,"numMessages":2,"numTools":68}
{"n":2,"t":"…15:13:11.529Z","mode":"clean","trigger":true,"numMessages":2,"numTools":68}

The TUI shows only the successful retry response; the failed attempt emits nothing. Session JSONL: 0 protocol tags, 0 leaked payload fragments, only the clean text persisted.

after: only the clean retried response is rendered

3. Regression check — ordinary JSON answer passes through

Same payload without protocol tags (TRIGGER_JSON): delivered unchanged at the terminal event and persisted intact, no retry, single upstream call. The documented buffering tradeoff behaves as described.

regression: plain JSON array answer delivered unchanged

4. Probe — review round-3 finding 1 is reachable end-to-end at the PR head

I replayed the same leaked payload but with a tool_calls delta (id + name agent, arguments never completed) arriving mid-payload — i.e. a provider that starts a structured call, abandons it, and leaks the payload as text. At the PR head this escapes the guard in the real pipeline:

  • the full payload including </parameter></function> renders in the UI and is persisted (2 × </parameter> in the chat file), no retry;
  • as a bonus, the abandoned delta materializes as a real Agent {} tool call that executes and fails validation.

probe: preparation metadata mid-payload lets the leak escape and executes an empty Agent call

This is the premature-releaseJsonCandidate() path from finding 1 of the round-3 review, now confirmed through the real converter → pipeline → TUI stack rather than only against the extracted detector. (The !hasToolCall ordering guard also applies here by design, but the user-visible outcome is the same leak this PR sets out to prevent.)

Unit tests

packages/core/src/core/geminiChat.test.ts at the PR head: 279/279 passed (24.1 s), including the new leak/regression/ordering cases.

Verdict

The fix works end-to-end for the exact #8207 production signature and does not regress ordinary JSON answers — the core claim of the PR holds in a real build. However, the probe in §4 shows the round-3 finding 1 gap is not theoretical: one interleaved preparation chunk re-opens the exact leak this PR closes. I'd still like that addressed (or explicitly scoped out with a code comment + follow-up issue) before merge.

中文版本

本地真实环境验证(合并参考)

macOS(Node v24)全链路真实验证,产品内部无任何 mock:两个独立 detached worktree——修复前 = PR base a4c0632c420,修复后 = PR head ca111fc8c47——各自 npm ci + 完整 npm run bundle,以真实交互式 TUI 运行(tmux 120×40 中 node dist/cli.js,隔离 $HOME,OPENAI_* 环境变量认证)。上游是本地 OpenAI 兼容 SSE 服务器,回放 #8207 生产特征:reasoning_content 思考 → 双 subagent JSON 数组分 4 个 content 块 → \n</parameter>\n</function>\nfinish_reason=stop + usage;重试请求返回干净响应。所有上游请求记录为 JSONL;会话持久化检查 ~/.qwen/projects/*/chats/*.jsonl。截图、mock 服务器、运行脚本和原始请求日志都在 pr-assets/8301-verify 分支,可复现。

1. 修复前(base a4c0632c420)——泄漏复现

仅 1 次上游调用,无重试。JSON payload 协议结束标签直接渲染为助手正文,并持久化到会话 JSONL(chat 文件中 2 处 </parameter>)。见上方第 1 张截图。

2. 修复后(PR head ca111fc8c47)——#8207 特征已修复

请求日志显示重试生效——同一会话 2.3 秒内发送两次(泄漏尝试 → 重试):n=1 leak、n=2 clean,消息数与工具数完全一致。TUI 只显示重试成功的响应,失败轮次零输出。会话 JSONL:0 协议标签、0 泄漏片段,只持久化了干净文本。见第 2 张截图。

3. 回归检查——普通 JSON 回答原样放行

相同 payload 但不带协议标签(TRIGGER_JSON):在终止事件时原样交付并完整持久化,无重试,单次上游调用。PR 描述的缓冲权衡行为与文档一致。见第 3 张截图。

4. 探针——round-3 审查 finding 1 在 PR head 端到端可达

回放相同泄漏 payload,但在 payload 中途插入一个 tool_calls delta(有 id + name agent,参数永不补全)——即 provider 先启动结构化调用、随后放弃并把参数以文本泄漏。在 PR head 的真实流水线中泄漏逃逸:

  • 完整 payload 连同 </parameter></function> 渲染进 UI 并被持久化(chat 文件 2 处 </parameter>),无重试;
  • 额外地,被放弃的 delta 物化为真实的 Agent {} 工具调用,实际执行并在参数校验处失败。

这正是 round-3 审查 finding 1 的 releaseJsonCandidate() 提前放行路径,如今经真实 converter → pipeline → TUI 全链路确认,而非仅在抽取的检测器上复现。(!hasToolCall 顺序保护在此场景同样生效,属设计行为,但用户可见结果就是本 PR 要阻止的那种泄漏。)见第 4 张截图。

单元测试

PR head 上 packages/core/src/core/geminiChat.test.ts:279/279 通过(24.1 秒),含新增泄漏/回归/顺序用例。

结论

针对 #8207 的确切生产特征,修复在真实构建中端到端生效,且不回归普通 JSON 回答——PR 的核心主张成立。但 §4 探针表明 round-3 finding 1 的缺口并非理论问题:一个交错的 preparation 块就能重新打开本 PR 要关闭的泄漏。建议合并前解决(或以代码注释 + 后续 issue 显式排除范围)。

@yiliang114 yiliang114 added the autofix/takeover Summon the autofix loop to manage this PR (remove to release; needs triage+) label Aug 1, 2026
@wenshao

wenshao commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Re-verification at 95f7e0cebf6 — all clear, including the finding-1 probe

Same real-stack harness as my previous run (fresh detached worktree, npm ci + full npm run bundle, real TUI in tmux with isolated $HOME, local OpenAI-compatible SSE mock, per-request JSONL logging, session-persistence checks), re-run against the new head.

1. #8207 production signature — still fixed

Two upstream calls 2.3 s apart (leak attempt → retry), only the clean response rendered and persisted, 0 protocol tags in the session JSONL.

{"n":1,"t":"…16:18:15.087Z","mode":"leak","trigger":true,"numMessages":2,"numTools":68}
{"n":2,"t":"…16:18:17.389Z","mode":"clean","trigger":true,"numMessages":2,"numTools":68}

new head: leak retried, only clean response rendered

2. Ordinary JSON answer — still passes through

Single upstream call, the JSON array delivered at the terminal event and persisted unchanged, no retry.

new head: plain JSON array delivered unchanged

3. Finding-1 probe — now fixed

Same probe that leaked at ca111fc8c47: a tool_calls delta (id + name, arguments never completed) arriving mid-payload, then the payload leaking as text with closing tags. At 95f7e0cebf6 the outcome flips to the correct behavior:

  • retry fires exactly like the plain case (2 upstream calls, 2.3 s apart, identical request shape);
  • nothing from the failed attempt reaches the UI or the session JSONL (0 protocol tags, 0 payload fragments);
  • the abandoned delta no longer materializes as an empty Agent {} execution.

new head: preparation-interrupted leak now retried cleanly

This matches the implementation change: releaseJsonCandidate() and both of its mid-payload call sites are gone, preparation-only chunks pass through without finalizing the buffer, and the new end-of-stream block releases (or drops, when leaked) buffered parts when a tool call is present — which also closes the finding-3 "buffered text silently dropped when hasToolCall is true" sub-case. The two new it.each tests pin exactly these paths.

Unit tests

geminiChat.test.ts at 95f7e0cebf6: 283/283 passed.

New evidence and raw request logs appended to pr-assets/8301-verify.

Verdict

My round-3 blocking finding is resolved and verified end-to-end through the real converter → pipeline → TUI stack. From my side this is good to merge.

中文版本

95f7e0cebf6 复验——全部通过,含 finding-1 探针

上一轮相同的真实栈验证环境(全新 detached worktree、npm ci + 完整 npm run bundle、tmux 真实 TUI + 隔离 $HOME、本地 OpenAI 兼容 SSE mock、逐请求 JSONL 日志、会话持久化检查),对新 head 重跑。

1. #8207 生产特征——仍然修复

2.3 秒内两次上游调用(泄漏尝试 → 重试),只渲染和持久化干净响应,会话 JSONL 中 0 协议标签。见第 1 张截图。

2. 普通 JSON 回答——仍原样放行

单次上游调用,JSON 数组在终止事件时交付并完整持久化,无重试。见第 2 张截图。

3. Finding-1 探针——已修复

与在 ca111fc8c47 上泄漏的同一探针(payload 中途插入参数永不补全的 tool_calls delta,随后 payload 带结束标签以文本泄漏),在 95f7e0cebf6 上行为翻转为正确:

  • 重试与普通场景完全一致(2 次上游调用、间隔 2.3 秒、请求形态相同);
  • 失败轮次的任何内容都未进入 UI 或会话 JSONL(0 协议标签、0 payload 片段);
  • 被放弃的 delta 不再物化为空参数的 Agent {} 执行。

与实现改动吻合:releaseJsonCandidate() 及其两个 payload 中途调用点已删除,仅含 preparation 的块直接放行且不再终结缓冲,新增的流末尾逻辑在存在 tool call 时补发(泄漏时丢弃)缓冲 parts——这也顺带关闭了 finding-3 中"hasToolCall 为 true 时缓冲文本被静默丢弃"的子问题。新增的两组 it.each 测试正好钉住这些路径。见第 3 张截图。

单元测试

95f7e0cebf6geminiChat.test.ts:283/283 通过

新证据与原始请求日志已追加至 pr-assets/8301-verify

结论

我 round-3 的阻断性 finding 已解决,并经真实 converter → pipeline → TUI 全链路端到端验证。就我而言可以合并。

@qwen-code-dev-bot

Copy link
Copy Markdown
Collaborator

🤝 Takeover engaged: the autofix loop now manages this PR — it will address new review feedback and resolve base conflicts until the label is removed or the round cap is reached. Remove the autofix/takeover label (or comment @qwen-code /takeover stop) to release.

中文说明

🤝 已接管:autofix 循环现在管理此 PR —— 将持续处理新的评审反馈与 base 冲突,直到移除标签或达到轮次上限。移除 autofix/takeover 标签(或评论 @qwen-code /takeover stop)即可释放。

@qwen-code-dev-bot

qwen-code-dev-bot commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

AutoFix round 3 finishedview run. See this round's report below.

中文说明

AutoFix 第 3 轮已完成 —— 查看运行。本轮报告见下方。

@qwen-code-dev-bot

Copy link
Copy Markdown
Collaborator

🤖 Reviewed the latest feedback — no changes needed. Why, point by point: · 已审阅最新反馈——无需改动。逐点说明原因如下:

Autofix review round — no code change

This round triaged the feedback newer than the last evaluation. The issue-level
review thread has concluded: the reviewer re-verified the current head
(95f7e0ceb) end-to-end through the real converter → pipeline → TUI stack and
reported "good to merge," and there are no failed or still-red CI checks.

The only actionable inline finding is a single automated-reviewer Suggestion
(packages/core/src/core/geminiChat.ts:1194): the escape-handling branch
(escaped flag) in hasLeakedToolCallTags has no dedicated test coverage — a
concrete surviving mutant. I verified against the code that the suggestion is
technically accurate (the only string-interior test, does not reject closing protocol tags inside a JSON string, uses a value with no backslash escapes, so
the escaped branch is never exercised) and that the implementation itself is
correct.

Disposition: deferred (no code change). This is a non-blocking, test-only
coverage gap with the production code confirmed correct. The PR has already been
through more than five review/fix rounds, so under the repository's review policy
only Critical fixes (correctness, security, data loss, regressions) land at this
stage and remaining Suggestions are deferred to a follow-up. The maintainer
(@yiliang114) has also explicitly recorded this exact deferral in the thread,
preferring not to reset the full review and CI loop for a mutation-only test. The
finding thread is left open and a reply records the disposition so the gap stays
visible for a follow-up.

No source files were modified and no commit was created this round.

中文说明

Autofix 审查轮次——无代码改动

本轮分诊了上次评估之后的新反馈。Issue 级审查线程已结束:审查者已对当前 head(95f7e0ceb)经真实 converter → pipeline → TUI 全链路做了端到端复验,并给出「可以合并」的结论,且没有失败或持续飘红的 CI 检查。

唯一可执行的行内 finding 是一条自动化审查器的 Suggestion(packages/core/src/core/geminiChat.ts:1194):hasLeakedToolCallTags 中的转义处理分支(escaped 标志)缺少专门的测试覆盖——存在一个具体存活的变异体。我已对照代码核实:该建议在技术上准确(唯一的字符串内部测试 does not reject closing protocol tags inside a JSON string 使用的值不含反斜杠转义,因此 escaped 分支从未被执行),且实现本身是正确的。

处置:延迟处理(无代码改动)。 这是一个非阻断、纯测试的覆盖缺口,生产代码已被确认正确。本 PR 已经历超过五轮审查/修复,因此按照仓库的审查策略,此阶段只落地 Critical 修复(正确性、安全性、数据丢失、回归),其余 Suggestion 延迟到后续处理。维护者(@yiliang114)也已在该线程中明确记录了同样的延迟决定,不希望为一个纯变异测试而重置完整的审查与 CI 循环。该 finding 线程保持打开,并附一条回复记录处置结论,以便该覆盖缺口在后续工作中保持可见。

本轮未修改任何源文件,也未创建任何提交。

Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。


🧠 Handled by Qwen Code · model/模型 qwen3.8-max-preview

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed. Suggestions are inline. Not reviewed: This PR extends LeadingProtocolTagLeakDetector in package... — the agent made no tool call: it read nothing.

中文说明

已审查。 建议见行内评论。 未审查:This PR extends LeadingProtocolTagLeakDetector in package...——该 agent 未发起任何工具调用:它什么都没读。

— qwen3.8-max-preview via Qwen Code /review

Comment thread packages/core/src/core/geminiChat.ts Outdated
Comment on lines +1197 to +1200
} else if (
(char === '}' || char === ']') &&
LEAKED_TOOL_CALL_TAGS.test(text.slice(i))
) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] hasLeakedToolCallTags calls text.slice(i) for every structural } / ] outside a JSON string, allocating a fresh O(n−i) substring on each iteration — the scan is quadratic in the buffered response length. — Concrete cost: this runs from finish() precisely when a response starts with { / [{ (the common structured-output shape that this diff newly buffers). A large minified JSON response (~256KB, ~30K top-level braces) triggers ~30K substring allocations; a probe measured ~3MB of temporary allocation on a 1.26MB / 100K-brace input versus 0MB for a sticky-regex variant. Wall-clock impact is modest today (V8 collects the short-lived slices efficiently), but the allocation work is avoidable. A sticky regex (drop ^, add the y flag, set lastIndex = i) or a startsWith forward scan makes the whole function O(n) with zero intermediate allocations, e.g.:

const LEAKED_TOOL_CALL_TAGS = /[}\]]\s*<\/parameter>\s*<\/function>/iy;
// ...
} else if (char === '}' || char === ']') {
  LEAKED_TOOL_CALL_TAGS.lastIndex = i;
  if (LEAKED_TOOL_CALL_TAGS.test(text)) return true;
}
中文说明

hasLeakedToolCallTags 对 JSON 字符串外的每个结构字符 } / ] 都调用 text.slice(i),每次分配一个 O(n−i) 的新子串,使扫描相对于缓冲响应长度呈平方复杂度。— 具体代价:该函数仅在响应以 { / [{ 开头时(即常见的结构化输出形态,本 diff 新增的缓冲逻辑)由 finish() 调用。一个较大的压缩 JSON 响应(约 256KB、约 3 万个顶层花括号)会触发约 3 万次子串分配;探针实测在 1.26MB / 10 万个花括号的输入上产生约 3MB 临时分配,而 sticky 正则变体为 0MB。当前墙钟时间影响不大(V8 能高效回收这些短命子串),但这部分分配工作是可以避免的。改用 sticky 正则(去掉 ^、加 y 标志、设置 lastIndex = i)或 startsWith 前向扫描,可使整个函数变为 O(n) 且零中间分配。

— qwen3.8-max-preview via Qwen Code /review

Comment on lines +4172 to +4174
const parts = takePendingProtocolParts();
if (parts.length > 0) {
content = {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] The non-leaked branch of the in-loop finishReason && !content?.parts path is not exercised by any test. — Concrete cost: the only test that sends a finish-only chunk ({ candidates: [{ finishReason: 'STOP' }] }) does so inside the leak test, where the detector is already in the leaked state, so only the pendingProtocolParts = [] branch runs. The else branch here — takePendingProtocolParts() plus the role: content?.role ?? 'model' content reconstruction — has no coverage. A future change that broke this branch would silently drop buffered, non-leaked JSON text from both the emitted stream and getHistory(), with no failing test. Consider adding a test that streams a JSON array without leaked tags across chunks, then a finish-only chunk, and asserts the JSON text appears in the emitted CHUNK events and in history.

中文说明

循环内 finishReason && !content?.parts 路径的“未泄漏”分支没有任何测试覆盖。— 具体代价:唯一发送仅含 finish reason 的数据块({ candidates: [{ finishReason: 'STOP' }] })的测试位于泄漏测试中,此时检测器已处于 leaked 状态,因此只会走到 pendingProtocolParts = [] 分支。这里的 else 分支——takePendingProtocolParts() 加上 role: content?.role ?? 'model' 的 content 重建——没有覆盖。如果未来的改动破坏了该分支,缓冲的、未泄漏的 JSON 文本会被静默地从输出流和 getHistory() 中丢弃,且没有任何测试失败。建议新增一个测试:跨多个数据块流式返回不带泄漏标签的 JSON 数组,随后发送一个仅含 finish reason 的数据块,并断言该 JSON 文本出现在输出的 CHUNK 事件以及 history 中。

— qwen3.8-max-preview via Qwen Code /review

@qwen-code-dev-bot

Copy link
Copy Markdown
Collaborator

🤖 Addressed the latest review feedback (round 1/100). What changed, and what I pushed back on: · 已处理最新评审反馈(第 1/100 轮)。改动内容与我反驳保留之处如下:

Review feedback addressed

Feedback points

1. [rc:3696128433] Quadratic allocation in hasLeakedToolCallTagsImplemented

Suggestion: Replace text.slice(i) per structural }/] with a sticky regex (y flag, lastIndex = i) to make the scan O(n) with zero intermediate allocations.

Change: Dropped the ^ anchor from LEAKED_TOOL_CALL_TAGS and added the y (sticky) flag. The call site now sets lastIndex = i and tests against the full text instead of allocating a substring. Semantically identical — the sticky flag anchors the match at lastIndex just as ^ anchored at position 0 of the old slice.

Files: packages/core/src/core/geminiChat.ts

2. [rc:3696128439] Missing test for non-leaked finish-only branch — Implemented

Suggestion: Add a test that streams JSON across chunks without leaked tags, then sends a finish-only chunk (finishReason with no content.parts), and asserts the JSON text appears in emitted CHUNK events and in history.

Change: Added releases buffered JSON through a finish-only chunk without leaked tags test. It streams a JSON array in two text chunks (no finish reason), then a finish-only chunk, and asserts: no retry, emitted text equals the original JSON, getLastModelMessageText() matches, and getHistory().at(-1)?.parts equals the emitted parts. This exercises the previously uncovered else branch of the finishReason && !content?.parts path (takePendingProtocolParts() + content reconstruction).

Files: packages/core/src/core/geminiChat.test.ts

Conflict notes

No conflicts (--conflict false).

Verification

  • npx tsc --noEmit (packages/core) — passed
  • npx eslint packages/core/src/core/geminiChat.ts packages/core/src/core/geminiChat.test.ts — passed (no errors)
  • npx vitest run src/core/geminiChat.test.ts (packages/core) — 284 tests passed
中文说明

已处理的审查反馈

反馈要点

1. [rc:3696128433] hasLeakedToolCallTags 中的二次分配 — 已实现

建议: 将每个结构字符 }/] 处的 text.slice(i) 替换为 sticky 正则(y 标志,lastIndex = i),使扫描变为 O(n) 且零中间分配。

变更:LEAKED_TOOL_CALL_TAGS 中移除 ^ 锚点并添加 y(sticky)标志。调用处现在设置 lastIndex = i 并对完整 text 进行测试,而非分配子串。语义完全相同——sticky 标志在 lastIndex 处锚定匹配,正如 ^ 曾在旧 slice 的位置 0 处锚定一样。

文件: packages/core/src/core/geminiChat.ts

2. [rc:3696128439] 缺少未泄漏 finish-only 分支的测试 — 已实现

建议: 新增一个测试:跨多个数据块流式返回不带泄漏标签的 JSON,随后发送一个仅含 finish reason 的数据块(finishReason 但无 content.parts),并断言 JSON 文本出现在输出的 CHUNK 事件以及 history 中。

变更: 新增测试 releases buffered JSON through a finish-only chunk without leaked tags。该测试将 JSON 数组分两个文本数据块流式发送(无 finish reason),然后发送一个仅含 finish reason 的数据块,并断言:无重试、输出文本等于原始 JSON、getLastModelMessageText() 匹配、且 getHistory().at(-1)?.parts 等于输出的 parts。这覆盖了此前未测试的 finishReason && !content?.parts 路径的 else 分支(takePendingProtocolParts() + content 重建)。

文件: packages/core/src/core/geminiChat.test.ts

冲突说明

无冲突(--conflict false)。

验证

  • npx tsc --noEmit(packages/core)— 通过
  • npx eslint packages/core/src/core/geminiChat.ts packages/core/src/core/geminiChat.test.ts — 通过(无错误)
  • npx vitest run src/core/geminiChat.test.ts(packages/core)— 284 个测试全部通过

Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。

Re-review when you have a moment. After round 100 this bot stops and leaves the PR for a human. · 有空请复审;第 100 轮后本 bot 停止并将 PR 交给人工。


🧠 Handled by Qwen Code · model/模型 qwen3.8-max-preview

@wenshao

wenshao commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Review (round 4) — fix(core): retry leaked JSON tool protocol output

Verified locally on head 4a374ef32 in a detached worktree: packages/core/src/core/geminiChat.test.ts passes 284/284. I also re-extracted LeadingProtocolTagLeakDetector plus the processStreamResponse pending-part loop into a standalone harness for state-machine probing, and ran three ad-hoc integration probes through the real sendMessageStream.

Delta since my last pass (95f7e0ceb4a374ef32)

Only two changes, both from the autofix round:

  1. Sticky regex in hasLeakedToolCallTags (geminiChat.ts:1184, :1197-1200) — ^... on text.slice(i) replaced with /…/iy + lastIndex = i. Correct and equivalent. Sticky anchors at lastIndex exactly as ^ anchored at position 0 of the old slice, test() on a failed sticky match resets lastIndex to 0, and every call site re-assigns lastIndex before testing, so no stale state can leak between iterations or calls. The function is synchronous and non-recursive, so the module-level mutable regex is safe. Verified identical verdicts on 8 shapes (production signature, in-string tags, escaped-quote payloads, uppercase tags, plain JSON).
  2. releases buffered JSON through a finish-only chunk without leaked tags — closes the previously uncovered else branch of finishReason && !content?.parts. Replayed the same sequence through my harness: two text chunks + a finish-only chunk release the JSON intact with pending drained to 0. Good test.

Nothing else moved. No critical issues in the delta — good to merge from my side.

Full-diff re-check

I re-walked the pending/release invariants once more since the buffering path is the risky part:

  • Buffer ⟹ pending invariant holds. Every path where accept() returns '' also pushes the part onto pendingProtocolParts, so outputParts.push(...takePendingProtocolParts(), part) pushing the original part (not {...part, text}) is correct — the released buffer text is already represented by the pending parts. Confirmed with a split-Markdown-link case ([ / link](url) rest) round-tripping to [link](url) rest with no duplication.
  • All four release sites converge. finish-only chunk, finish-on-content chunk, post-loop tool-call drain, and the leaked-drop path all leave pending empty; the leak throw at :4477 runs after the post-loop drain, so a released turn can't also throw.
  • Empty-string deltas (content: '', common on OpenAI-compatible SSE) route through pendingProtocolParts but recover on the next non-empty text part or at the finish chunk — verified no text loss for leading/mid/trailing empty deltas.
  • isToolCallPreparationOnlyhasCandidateOutput is a semantic widening, not a rename: usage-only chunks no longer set streamYieldedAnyChunk, so model fallback stays eligible after usage metadata. That's the intended fix and it's pinned by the updated fallback test; the only remaining use of streamYieldedChunk is a debug log field, so blast radius is contained.

Non-blocking follow-ups (do NOT reset this PR)

1. Detection is brittle to near-miss shapes of the same leak. LEAKED_TOOL_CALL_TAGS requires } or ] immediately before the tags, so a leak whose argument object was never closed slips through. Probed through the real sendMessageStream on this head:

leaked text retried?
{"file_path": "a.ts"</parameter></function> ❌ no
{"file_path":"a.ts",</parameter></function> ❌ no
[{"file_path":"a.ts"}]</function> ❌ no

All three are emitted and persisted as assistant text — the exact failure mode #8207 describes, one truncated brace away. This is not a regression (pre-PR they leaked too) and the production signature is covered, so it shouldn't block. For the follow-up: once the detector has committed to the json state, any </parameter> or </function> occurring outside a JSON string is already unambiguous — valid JSON can't contain bare text between tokens — so the closing-brace anchor can be dropped entirely without raising false-positive risk. The existing quote/escape scanner already gives you the "outside a string" test for free.

2. No diagnostics on the suppression path. When a leak is detected the buffered parts — sometimes including a real functionCall, per the retries when a function call interrupts a partial JSON protocol leak test — are dropped with no debugLogger line, while the neighbouring XML-fallback path logs both success and rejection. A one-line debugLogger.warn with buffer length and whether a functionCall was discarded would make this diagnosable from a user's session log instead of requiring a repro.

3. The subtle bits deserve comments. In a file that carries multi-paragraph design notes on every other non-obvious branch, pendingProtocolParts.push(...outputParts.splice(0), part) and the "push part, not {...part, text}" asymmetry are the two lines a future reader is most likely to "simplify" into a duplication bug. Two short comments would pin the invariant.

4. escaped-branch coverage — already triaged and deferred in this thread; noting it only so the follow-up carries all four items together.

Known tradeoff (already documented, no action)

A response whose first non-whitespace text is { or [{ is buffered end-to-end and delivered as a single part at the terminal event, with all intermediate chunks suppressed. For a "reply with JSON only" prompt that means no incremental rendering for the whole response and an unbounded in-memory buffer. The PR body calls this out explicitly and the production payload is only ~905 bytes, so it's the right call for now; if it ever bites, a size cap (release past a few KB — well above any realistic leaked argument blob) is a name-agnostic mitigation that preserves the current detection.

Verdict

Approve. The delta is a clean, correct refactor plus a genuine coverage gain, the full suite is green, and the four items above are follow-up material, not merge blockers.

中文说明

第 4 轮评审

4a374ef32 上于独立 worktree 本地验证:geminiChat.test.ts 284/284 通过。另将 LeadingProtocolTagLeakDetectorprocessStreamResponse 的 pending-part 循环抽到独立 harness 做状态机探测,并通过真实 sendMessageStream 跑了 3 个即席集成探针。

与上次评审(95f7e0ceb)的增量

  1. sticky 正则^ + slice(i) 改为 /…/iy + lastIndex = i语义等价且正确。失败的 sticky test() 会把 lastIndex 复位为 0,且每次调用前都重新赋值,不存在状态残留;函数同步且不递归,模块级可变正则安全。8 种形态验证结论一致。
  2. finish-only 分支的新测试:覆盖了此前未测的 finishReason && !content?.partselse 分支,harness 复放确认 JSON 完整释放、pending 归零。

增量部分无 Critical 问题,可以合并

全量复查

  • 「buffer 非空 ⟹ pending 非空」不变量成立,因此释放时 push 原始 part(而非 {...part, text})是正确的,不会重复输出(已用拆分的 Markdown 链接用例验证)。
  • 四处释放点均会把 pending 清空;泄漏抛出点在 post-loop drain 之后,不会既释放又抛出。
  • 空字符串 delta(OpenAI 兼容 SSE 常见)会暂存但在下一个非空文本或 finish 处恢复,无文本丢失。
  • isToolCallPreparationOnlyhasCandidateOutput 是语义扩大而非纯重命名:usage-only chunk 不再置位 streamYieldedAnyChunk,从而不阻断 fallback——这正是本次意图,且已有测试固定。

非阻断的后续项(不要为此重置本 PR)

  1. 检测对同类泄漏的近似形态很脆弱:正则要求标签前紧邻 }/],因此参数对象未闭合的泄漏会漏检。真实链路探针结果:{"file_path": "a.ts"</parameter></function>{"file_path":"a.ts",</parameter></function>[{"file_path":"a.ts"}]</function> 三者均不重试,直接作为文本输出并持久化。这不是回归(改前同样泄漏),生产特征也已覆盖,故不阻断。后续可考虑:进入 json 状态后,出现在 JSON 字符串之外</parameter></function> 本身就已无歧义(合法 JSON 的 token 之间不可能有裸文本),可直接去掉闭合括号锚点而不增加误报;现有的引号/转义扫描已经提供了「是否在字符串内」的判定。
  2. 抑制路径没有任何日志:泄漏时丢弃的 pending parts 有时包含真实 functionCall,却没有 debugLogger 记录,而相邻的 XML fallback 路径成功与拒绝都会记日志。建议补一行 warn(buffer 长度 + 是否丢弃了 functionCall)。
  3. 两处易被「优化」成 bug 的代码建议加注释pendingProtocolParts.push(...outputParts.splice(0), part) 与「push part 而非 {...part, text}」的不对称。
  4. escaped 分支覆盖:本线程已决定延后,仅一并列出便于后续统一处理。

已知权衡(已在 PR 描述中说明,无需处理)

首个非空白文本为 {[{ 的响应会整体缓冲到终止事件才一次性输出,期间所有 chunk 被抑制。对「只输出 JSON」类提示意味着全程无增量渲染,且缓冲无上限。PR 已显式记录该权衡,生产 payload 仅约 905 字节,当前取舍合理;若日后成为问题,加一个体积上限(超过数 KB 即释放)是与参数名无关且不影响现有检测的缓解手段。

结论

同意合并。 增量是干净正确的重构加一处真实覆盖增益,全量测试通过,上述四项属后续跟进而非合并阻断。

@qwen-code-dev-bot

Copy link
Copy Markdown
Collaborator

🤖 Reviewed the latest feedback — no changes needed. Why, point by point: · 已审阅最新反馈——无需改动。逐点说明原因如下:

No action required — PR approved

The round-4 review from @wenshao is an explicit APPROVE on head 4a374ef32:

  • No inline comments or CHANGES_REQUESTED items.
  • No failed CI checks.
  • The four non-blocking follow-ups (brittle near-miss detection shapes, missing suppression-path diagnostics, two comment-worthy invariants, escaped-branch coverage) are explicitly marked as "follow-up material, not merge blockers" and "do NOT reset this PR."

No code changes are warranted this round.

中文说明

无需操作 — PR 已获批准

第 4 轮评审(@wenshao,基于 4a374ef32)明确给出 同意合并 的结论:

  • 无行内评论,无 CHANGES_REQUESTED 项。
  • 无失败的 CI 检查。
  • 四项非阻断后续项(近似形态漏检、抑制路径缺少日志、两处值得加注释的不变量、escaped 分支覆盖)均被明确标注为"后续跟进而非合并阻断"且"不要为此重置本 PR"。

本轮无需任何代码变更。

Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。


🧠 Handled by Qwen Code · model/模型 qwen3.8-max-preview

@doudouOUC doudouOUC left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Downgraded from Approve to Comment: CI still running. Reviewed.

— qwen3.7-max via Qwen Code /review

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed. Suggestions are inline. Not reviewed: build-and-test — Integration Tests (CLI, No Sandbox) was skipped in CI and its suite did not run locally.

中文说明

已审查。 建议见行内评论。 未审查:build-and-test — Integration Tests (CLI, No Sandbox) was skipped in CI and its suite did not run locally。

— qwen3.8-max-preview via Qwen Code /review (v0.21.3)

Comment on lines +4343 to +4345
protocolTagDetector.finish();
if (protocolTagDetector.leaked) {
pendingProtocolParts = [];

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] The leaked branch of this post-stream synthetic-chunk block is not exercised by any test. — Concrete cost: a stream that emits leaked JSON (an object array followed by the closing parameter-end / function-end protocol tags) plus a functionCall part, and then ends with no finishReason, reaches this block as the only code path that calls finish() and clears the leak. Probe-confirmed: neutralizing the if (protocolTagDetector.leaked) { pendingProtocolParts = []; } guard ships the leaked protocol tags to the user instead of retrying, and the full 284-test suite still passes with that mutant — no existing test discriminates this branch. The closest test, "retries leaked JSON before a structured tool call", sends a finish reason, so it exercises the in-loop finishReason guard rather than this post-stream one. Suggested fix: add a case mirroring "preserves leading JSON when a tool call ends without a finish reason" but with leaked protocol tags in the buffered JSON and no finish reason, asserting that a retry occurs and no leaked text is emitted.

中文说明

[Suggestion] 这段「流结束后合成 chunk」代码块中的 leaked 分支没有任何测试覆盖。— 具体代价:当一个流先输出泄漏的 JSON(一个对象数组,后跟 parameter 结束标签 / function 结束标签这类协议标签)外加一个 functionCall part,然后在没有 finishReason 的情况下结束时,只有这个代码块会调用 finish() 并清除泄漏。经探针验证:把 if (protocolTagDetector.leaked) { pendingProtocolParts = []; } 这个保护置为无效后,泄漏的协议标签会直接输出给用户而不是触发重试,并且整套 284 个测试在该 mutant 下仍然全部通过——没有任何现有测试能区分这个分支。最接近的测试 "retries leaked JSON before a structured tool call" 会发送 finish reason,因此它走的是循环内的 finishReason 保护,而非这个流结束后的分支。建议修复:参照 "preserves leading JSON when a tool call ends without a finish reason" 增加一个用例,但让缓冲的 JSON 中带泄漏的协议标签且不带 finish reason,断言会触发重试且不输出任何泄漏文本。

— qwen3.8-max-preview via Qwen Code /review (v0.21.3)

@doudouOUC doudouOUC left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — fix(core): retry leaked JSON tool protocol output

Verdict: COMMENT (downgraded from Approve: CI still running)

Summary

This PR extends LeadingProtocolTagLeakDetector in geminiChat.ts to catch a production variant where the model serializes tool-call arguments as plain-text JSON followed by leaked protocol close tags. The failed attempt is routed through the existing protocol-leak retry path. Detection stays at the shared stream boundary, buffering a leading JSON object or object array without guessing argument names, preserving part ordering while ambiguous, and releasing ordinary JSON or real structured tool-call events unchanged.

Review Details

  • 12 review agents launched in parallel (issue fidelity, line-by-line, removed-behavior, etc.)
  • Verification rejected both initial findings (JSON buffering delay is documented design decision; post-stream pending parts handled by existing NO_FINISH_REASON path)
  • Reverse audit: two consecutive dry rounds confirmed convergence
  • Coverage: 3/3 chunks, 15 agents, all did work
  • 0 confirmed findings, 0 inline comments

Conclusion

No blockers. The PR faithfully fixes #8207 with a well-tested approach (321 affected unit tests). The buffering tradeoff is acknowledged in the PR description. Approve once CI passes.

— qwen3.7-max via Qwen Code /review

@qwen-code-dev-bot

Copy link
Copy Markdown
Collaborator

🤖 Addressed the latest review feedback (round 2/100). What changed, and what I pushed back on: · 已处理最新评审反馈(第 2/100 轮)。改动内容与我反驳保留之处如下:

Review feedback addressed

[rc:3696470416] Post-stream leaked branch not exercised by any test — Implemented

The automated reviewer identified that the leaked branch of the post-stream synthetic-chunk block (the if (protocolTagDetector.leaked) { pendingProtocolParts = []; } guard) was not covered by any existing test. A mutation probe confirmed that neutralizing this guard ships leaked protocol tags to the user instead of retrying, and the full test suite still passes with that mutant.

Added a new test retries leaked JSON without a finish reason via the post-stream leak guard that mirrors the existing retries leaked JSON before a structured tool call test but sends the functionCall part in a separate chunk without a finishReason on any chunk. This forces the stream to end without triggering the in-loop finishReason guard, so the post-stream block is the only code path that calls finish() and detects the leak. The test asserts that a retry occurs (2 calls to generateContentStream) and no leaked text is emitted.

Changed files: packages/core/src/core/geminiChat.test.ts (+63 lines)

Verification

  • npm run build — passed
  • npm run typecheck — passed
  • npm run lint — passed
  • vitest packages/core src/core/geminiChat.test.ts — 285 passed (285 total, including the new test)
中文说明

已处理的审查反馈

[rc:3696470416] 流结束后的 leaked 分支没有任何测试覆盖 — 已实现

自动审查器发现,流结束后合成 chunk 代码块中的 leaked 分支(即 if (protocolTagDetector.leaked) { pendingProtocolParts = []; } 保护)没有任何现有测试覆盖。经变异探针验证:将该保护置为无效后,泄漏的协议标签会直接输出给用户而不是触发重试,并且整套测试在该 mutant 下仍然全部通过。

新增了一个测试 retries leaked JSON without a finish reason via the post-stream leak guard,该测试参照现有的 retries leaked JSON before a structured tool call 测试,但将 functionCall part 放在一个单独的 chunk 中,且所有 chunk 都不带 finishReason。这迫使流在不触发循环内 finishReason 保护的情况下结束,从而使流结束后的代码块成为唯一调用 finish() 并检测泄漏的路径。测试断言会触发重试(generateContentStream 被调用 2 次)且不输出任何泄漏文本。

变更文件: packages/core/src/core/geminiChat.test.ts(+63 行)

验证

  • npm run build — 通过
  • npm run typecheck — 通过
  • npm run lint — 通过
  • vitest packages/core src/core/geminiChat.test.ts — 285 个测试全部通过(包含新增测试)

Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。

Re-review when you have a moment. After round 100 this bot stops and leaves the PR for a human. · 有空请复审;第 100 轮后本 bot 停止并将 PR 交给人工。


🧠 Handled by Qwen Code · model/模型 qwen3.8-max-preview

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed. Not reviewed: build-and-test — Integration Tests (CLI, No Sandbox) was skipped in CI and its suite did not run locally.

中文说明

已审查。 未审查:build-and-test — Integration Tests (CLI, No Sandbox) was skipped in CI and its suite did not run locally。

— qwen3.8-max-preview via Qwen Code /review (v0.21.3)

@qwen-code-dev-bot

Copy link
Copy Markdown
Collaborator

🤖 Reviewed the latest feedback — no changes needed. Why, point by point: · 已审阅最新反馈——无需改动。逐点说明原因如下:

No action required

The latest review round (round 2) from the automated reviewer returned a COMMENT with zero findings. There are no inline comments, no failed checks, and no still-red checks to address.

The reviewer noted that build-and-test — Integration Tests (CLI, No Sandbox) was skipped in CI and not run locally. This is an informational coverage note, not a finding — the changed files (packages/core/src/core/geminiChat.ts and its test) are exercised by the unit test suite, which passed. No code change is warranted.

中文说明

无需操作

自动化审查者的最新审查轮次(第 2 轮)返回了一条 COMMENT(评论),包含零个发现。没有行内评论、没有失败的检查、也没有持续红色的检查需要处理。

审查者指出 build-and-test — Integration Tests (CLI, No Sandbox) 在 CI 中被跳过且未在本地运行。这是一条信息性的覆盖率说明,而非发现——变更的文件(packages/core/src/core/geminiChat.ts 及其测试文件)由单元测试套件覆盖,且单元测试已通过。无需进行代码变更。

Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。


🧠 Handled by Qwen Code · model/模型 qwen3.8-max-preview

@wenshao

wenshao commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

@qwen-code /triage

@qwen-code-ci-bot

qwen-code-ci-bot commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

Sandboxed verification: ❌ not passed — findings reported (agent verdict) - workflow run

Ran the PR in an isolated, token-free container: A/B against the base build, mock-free harness assertions, targeted gates. Advisory evidence for human reviewers — not a review, an approval, or a CI check.

Scripted assertions: 48 passed · 1 failed · 49 total

中文 — 判定:❌ 不通过 · 报告了发现(agent 判定)

沙箱验证在隔离、无凭证的容器中执行了该 PR 的代码(与 base 构建 A/B 对照、无 mock harness 断言、定向门禁)。仅作为评审证据,不构成评审、批准或 CI 检查

脚本断言:48 通过 · 1 失败 · 49 总计

Verification report

PR #8301 — deep verification report

Verdict: findings · verified head 45ed8f111 (merge base 184365390) · scripted assertions 48 pass / 1 fail / 49 total (assertions.json).

The central claim is decisively load-bearing: the A/B flips cleanly (01-ab-head-leak-suppressed.png vs 02-ab-base-leak-emitted.png) — at the base the production-shaped response from #8207 is emitted as assistant text and persisted to the recording with no retry, while at head it produces PROTOCOL_TAG_LEAK, emits zero parts from the failed attempt, leaves history/recording unchanged, retries once, and exposes only the successful response. The full geminiChat suite is 285 passed / 0 failed at head, and the central new test is non-vacuous (disabling the detector makes it fail on the behavioural mismatch, 04-vacuity-mutation.png). The single red assertion is a measured completeness finding (Finding 1, 06-boundary-head.png): one adjacent shape of the same leak bug-class — a prose sentence interleaved between the JSON payload and the leaked close tags — escapes the fix's anchored matcher. Per the publisher rule (nonzero fail ⇒ not merge-ready) and the skill rule that a fix holding only for the reported input shape is a finding, the verdict is findings. This is not a statement that the PR is broken: head is a strict superset of base's protection (base leaked every shape including this one; head leaks only this one), and a measured, behaviour-preserving candidate fix is provided.

Notation: the leaked protocol close tags are written below as </parameter> and </function> (entities, to keep this report's own markup intact). Verbatim leaked payloads are shown only in the evidence files (base-leak.json, the PNGs), never inlined here.

中文摘要
  • 结论findings。核心声明确为承重改动——A/B 对照发生翻转(01-ab-head-leak-suppressed.png02-ab-base-leak-emitted.png):base(HEAD^1)把 fix(core): JSON-style tool call arguments leak as plain text when model drops function-calling format #8207 的生产形态响应既作为助手文本输出、又写入 recording,且不重试;head 触发 PROTOCOL_TAG_LEAK,失败轮次输出 0 个 part,history/recording 不变,重试一次,仅暴露成功响应。geminiChat 全套件 285 通过 / 0 失败,且核心新测试非空(关闭检测器即因行为不符而失败,04-vacuity-mutation.png)。
  • 唯一红项 = 完整性 finding(Finding 1,06-boundary-head.png):同一泄漏 bug 类的一个相邻形态——JSON 参数与泄漏闭合标签之间夹了一句自然语言散文——逃过了该 fix 的「锚定到 }/] 之后」匹配器,在 head 仍被输出且不重试。依据「非零 fail ⇒ 不可 merge-ready」与「只对上报形态成立的修复即为 finding」两条规则,verdict 为 findings
  • 并非判定 PR 损坏:head 的保护是 base 的严格超集(base 连该形态也泄漏,head 仅漏此一种),且给出了已度量、行为保持的候选修复(07-boundary-broader.png)。
  • 未覆盖:逐 commit 归因(浅克隆 depth 2,仅 merge/base/head 可达);真实 provider 流量(按设计 mock HTTP 层——该缺陷是客户端流处理缺陷,mock 是正确 oracle 而非限制);全仓 build/typecheck/lint 与 321 受影响测试(仅跑了承重面 geminiChat 套件;更广门禁由 PR 自身 CI 负责)。详见 Not covered 与各表。

Central claim + A/B

Central claim. A model response that is a plain-text JSON object/object-array of tool arguments followed by leaked close tags </parameter></function> — with finish_reason=stop and no structured tool call — must not reach the UI, conversation history, or session recording; the failed attempt must route through the existing protocol-leak retry path.

Method (control validity). Base cell = git worktree add tmp/base-tree HEAD^1 (the merge-ref base tip). The PR touches only packages/core/src/core/geminiChat.{ts,test.ts}; git diff HEAD^1..HEAD --stat -- package.json package-lock.json packages/core/package.json is empty, so the dependency tree is unchanged and reusing the root node_modules is a clean control. The base worktree had no nested node_modules of its own (only a vite cache), so the 8 workspace-nested third-party deps (ajv v8, undici, …) were symlinked from head's packages/core/node_modules to satisfy vite's resolver; these are third-party packages, not @qwen-code/* internal links, and geminiChat.ts imports its siblings by relative path (../core/..., ../config/...), all of which resolve to the base source tree. The empirical A/B result is the proof the control loaded base code: base exhibited the leak, which only base source can produce. Identical harness (verify-leak.test.ts) drives the real GeminiChat.sendMessageStream with a mocked model HTTP layer in both trees; the only difference between cells is the source under test.

cell source generateContent calls RETRY seen leaked text emitted leaked text recorded emitted text
HEAD (fix) 45ed8f111 2 true false false Successful final response
BASE (control) 184365390 1 false true true the full leaked array + close tags (see base-leak.json)

Witnesses: 01-ab-head-leak-suppressed.png (head: callCount:2, retrySeen:true, leakedEmitted:false, leakedRecorded:false, test passed) and 02-ab-base-leak-emitted.png (base: callCount:1, retrySeen:false, leakedEmitted:true, leakedRecorded:true, with the verbatim leaked array + close tags in both emittedText and recordedText; the harness's fixed-behaviour assertion reds because the bug is present — that red is the expected control signal and is encoded as a passing assertion in assertions.mjs). The flip on leakedEmitted (head ≠ base) is the load-bearing proof.

Secondary claims (covered by the 285-test suite at head, not independently A/B'd — scope choice). (a) Non-leak shapes are preserved and numeric/markdown/prose keep streaming immediately — independently re-verified by the wire oracle below. (b) The isToolCallPreparationOnlyhasCandidateOutput rename and the new !chunk.candidates?.length || preparations.length > 0 yield-condition keep usage/preparation metadata flowing and do not block a later configured fallback — exercised by the suite's continues fallback after usage and preparation metadata and the tool-call-no-finish cases (all green). (c) The post-stream drain block (pendingProtocolParts flushed when a tool call is present and no leak) — exercised by the suite's preserves leading JSON when a tool call ends without a finish reason cases (green).

Corrections

None. This is a first verification round (no previous-report.md), and the PR description's own scope statement ("other malformed provider-specific tool syntaxes that do not match this production signature" are out of scope) is internally consistent — it is in fact the reason Finding 1 is rated Medium/completeness rather than Critical (see below). No inaccurate mechanism or misattributed cause was found in the PR text to correct.

Findings

F1 — Medium (completeness): a prose gap between the JSON payload and the leaked close tags escapes the detector

Severity rationale. The production repro from #8207 (JSON immediately followed by the close tags) is fixed and proven load-bearing above, so this is not a regression or a data-loss path. It is a completeness gap in a defence-in-depth guard: the same root cause (tool protocol rendered as plain text instead of a structured call) one shape down. The fix's own new tests pin the reported shape by construction; this sibling is exactly what they do not pin. It is rated Medium rather than Critical because (i) the immediate-tags shape is the documented production signature and is closed, (ii) the escape requires the model to interleave a natural-language sentence between the JSON args and the close tags — a less probable but plausible degeneration, and (iii) the PR's scope statement arguably carves it out. The sibling-sweep rule still requires surfacing it, and a clean fix exists.

Root cause. hasLeakedToolCallTags only tests the leaked close-tag regex at buffer positions where the character is } or ] (LEAKED_TOOL_CALL_TAGS = /[}\]]\s*<\/parameter>\s*<\/function>/iy). When a non-]/} character (a prose letter, space, etc.) sits between the JSON's structural close and the leaked tags, no anchor fires and the buffered text is released verbatim at finish().

Reproducing (head, clean tree). The boundary probe verify-sibling2.test.ts drives four shapes through the real stream path; the second escapes at head:

SIB2 {"label":"json+tags","retry":true,"tagEmitted":false}            # caught
SIB2 {"label":"json+prose+tags","retry":false,"tagEmitted":true}      # ESCAPES (finding)
SIB2 {"label":"json+tags+trailing","retry":true,"tagEmitted":false}   # caught
SIB2 {"label":"json+tag-in-string-value","retry":false,"tagEmitted":true}  # correct: literal tag inside a string value is legit content

(json+prose+tags = the leaked array, then \nHere is my note.\n, then the close tags.) Witness 06-boundary-head.png. The json+tag-in-string-value row is the important negative control: a legitimate JSON string value that literally contains the close tags must be emitted verbatim with no false retry — and it is, so any fix must preserve that in-string guard.

Blast radius. Same code path as the central fix (LeadingProtocolTagLeakDetector.finishrelease), so the consequence is identical in kind to #8207 — leaked protocol tags + raw arg JSON reach the UI, history, and recording without a retry — just gated on a less common interleave. No additional call sites; the detector is the single boundary.

Measured candidate fix (behaviour-preserving; not applied to the PR)

Broaden the matcher to fire on an out-of-string leaked close tag anywhere in the buffered JSON, not only immediately after }/]. The existing in-string scanner already skips quoted regions, so a tag inside a string value is still ignored. Scratch-copy change (regex + the one anchor branch in hasLeakedToolCallTags):

-const LEAKED_TOOL_CALL_TAGS = /[}\]]\s*<\/parameter>\s*<\/function>/iy;
+const LEAKED_CLOSE_TAG = /<\/(?:parameter|function)>/iy;
 ...
-    } else if (char === '}' || char === ']') {
-      LEAKED_TOOL_CALL_TAGS.lastIndex = i;
-      if (LEAKED_TOOL_CALL_TAGS.test(text)) return true;
+    } else if (char === '<') {
+      LEAKED_CLOSE_TAG.lastIndex = i;
+      if (LEAKED_CLOSE_TAG.test(text)) return true;
     }

Measured on the same harnesses (scratch copy, then restored — the PR tree was left pristine):

probe clean head head + candidate
json+prose+tags (the escape) retry:false, tagEmitted:true (leaks) retry:true, tagEmitted:false (caught)
json+tags, json+tags+trailing caught caught (unchanged)
json+tag-in-string-value (false-positive guard) emitted verbatim, no retry emitted verbatim, no retry (guard holds)
benign oracle (numeric/markdown/prose/objarray/toolcall) see head-shapes.jsonl byte-identical (diff empty)
full geminiChat suite 285 passed / 0 failed 285 passed / 0 failed

Witness 07-boundary-broader.png. So the candidate closes the gap with zero collateral on the unchanged shapes and the in-string guard, and the suite stays green. Because the suite is green on both sides, this candidate ships with its own pinning need: the PR's current suite has no fixture for the prose-gap shape, so add one (the json+prose+tags case above) alongside the matcher change — otherwise the next regression on this axis lands in an unpinned gap. I did not apply this to the PR (out of scope for verification, and the author may prefer a different matcher); it is offered as measured evidence that the gap is closable without regressions.

Not covered

  • Per-commit attribution. The CI checkout is depth 2 (git rev-parse --is-shallow-repository = true); only the merge commit, HEAD^1 (base tip) and HEAD^2 (PR head) are reachable. The snapshot's commits array lists 6 commits, but git rev-list HEAD^1..HEAD^2 cannot be trusted at a shallow boundary, so per-commit behaviour was not individually exercised — only the aggregate HEAD^1..HEAD diff was verified. The 6 commits are coherent (progressive hardening of one detector), so aggregate verification is adequate, but a per-commit table is not supported by the evidence.
  • Real provider traffic. The harness mocks the model HTTP layer by design — the defect under test is a client-side stream-processing bug, so the mock is the correct oracle (the wire bytes that trigger it are deterministic and reproduced exactly), not an environmental limitation. No live model call was made; behaviour against a specific provider's quirks beyond the reproduced signature is untested.
  • Repo-wide gates. I ran the load-bearing surface only: the geminiChat suite (285/0 at head). The PR's claim of "321 affected unit tests + full build/typecheck/ESLint/Prettier" is its own CI's responsibility and was not re-run here (the head tree was already built by the verify job's pre-step; I did not re-run npm run build/typecheck/lint repo-wide). The two changed files are confined to packages/core, so the omitted gates are low-risk for this diff, but they are unmeasured by this round.
  • Detector ladder / scaling. The new hasLeakedToolCallTags is a single linear scan with a sticky regex tested only at }/]/< positions; it is O(n) in buffer length with no nested quantifier over attacker-controlled repetition, so a hostile-input timing ladder was not run (no superlinear shape is present in the matcher). Noted rather than measured.
  • The hasCandidateOutput rename in isolation. Verified only via the suite (secondary claim b), not via a dedicated A/B cell; the rename is a pure predicate extraction plus a semantics change at the two call sites (it no longer requires "no preparations" to count as candidate output), and the suite's fallback/usage cases pin it.

Methodology

Environment: the CI verify container (node:22-bookworm), working tree = refs/pull/8301/merge at depth 2; npm ci + npm run build already complete at head. All harnesses are vitest specs that import the real GeminiChat and mock only the model HTTP layer + fs/telemetry/retry (the same boundary the PR's own tests use); they drive sendMessageStream and read the emitted StreamEvents and the recordAssistantTurn recording callback. The A/B base cell ran the identical verify-leak.test.ts in a scratch worktree at HEAD^1 wired to the head install (lockfile unchanged; nested third-party deps symlinked; no internal-package link crosses the boundary — asserted above and confirmed empirically by the base cell leaking). The wire oracle (verify-shapes3.test.ts) ran on both arms and emits one JSON line per shape; the comparator assertions.mjs reads the captured oracle files and compares each observed value to an encoded expectation (a missing file/shape fails its assertion, so a broken harness cannot fake green), encoding the base-arm reds and the vacuity red as the values those controls must show (counted as pass) and the prose-gap escape as the value the fix should produce (observed false ⇒ the one fail). Raw logs and per-cell JSON live in this directory (head-leak.json, base-leak.json, head-shapes.jsonl, base-shapes.jsonl, head-boundary.jsonl, gate-head.log, vacuity.log); the seven evidence PNGs were rendered from real runs via scripts/verify-capture.mjs. The full geminiChat suite was run twice on the clean head (285/0 both times) and once under each mutation (vacuity: 2 failed as required; broader-matcher: 285/0). After every mutation the source was restored and git diff --quiet HEAD -- packages/core/src/core/geminiChat.ts confirmed clean; the base worktree was removed.

Evidence images

01-ab-head-leak-suppressed

02-ab-base-leak-emitted

03-wire-oracle-shapes

04-vacuity-mutation

05-assertion-matrix

06-boundary-head

07-boundary-broader

Harness scripts and raw logs are in the workflow run artifacts (7-day retention).

Qwen Code · sandboxed verification

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, looks ready to ship. ✅

@qwen-code-dev-bot

Copy link
Copy Markdown
Collaborator

🤖 Reviewed the latest feedback — no changes needed. Why, point by point: · 已审阅最新反馈——无需改动。逐点说明原因如下:

No action required

This review round found no actionable feedback:

  • Reviews: none
  • Inline comments: none
  • Failed checks: none
  • Still-red checks: none

The only issue-level comment is an automated CI verification status notification, which requires no code change.

中文说明

无需操作

本轮审查未发现需要处理的反馈:

  • 代码审查:无
  • 行内评论:无
  • 失败的检查:无
  • 持续失败的检查:无

唯一的 issue 级别评论是自动化 CI 验证状态通知,无需进行代码更改。

Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。


🧠 Handled by Qwen Code · model/模型 qwen3.8-max-preview

@wenshao

wenshao commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

Local real-stack verification at 45ed8f111 — merge reference

Full end-to-end run on macOS (Node v24.18.1), no mocks inside the product. Two fresh detached worktrees off the GitHub merge ref, each npm ci + full npm run bundle, launched as the real interactive TUI (node dist/cli.js --yolo in tmux 120×40, isolated $HOME/$QWEN_HOME/$TMPDIR, auth via OPENAI_*). The only test double is upstream: a local OpenAI-compatible SSE server that replays the exact production shape from #8207 and logs every request.

commit contains hasLeakedToolCallTags
before 184365390 (merge base) no
after 26a99e433 = merge of 45ed8f111 into 184365390 yes (dist/chunks/chunk-3YWUEUAM.js)

The two trees differ only by this PR: git diff --stat 184365390 26a99e433geminiChat.ts + geminiChat.test.ts, 842 insertions / 34 deletions.

Result: the production bug reproduces on base and is fixed on head

Before — the leaked JSON payload and the </parameter> / </function> tags are rendered as an assistant message. One upstream request, no retry.

before

After — the failed attempt emits nothing, the turn is retried, and only the successful response reaches the UI.

after

Provider request log for the same prompt:

base:  1  A attempt=1  A/leaked-json
head:  1  A attempt=1  A/leaked-json
       2  A attempt=2  A/retry-clean     <-- retry only on head

Session persistence (~/.qwen/projects/<slug>/chats/<id>.jsonl), read back from disk:

base  assistant parts: [thought] | "[{\"name\":\"create_node\", ... }]\n\n</parameter>\n</function>\n"
      file contains "</parameter>":  true
      file contains "create_node":   true

head  assistant parts: "RETRY-OK: dispatching the two subagents through the tool channel."
      file contains "</parameter>":  false
      file contains "create_node":   false

So the leak is gone from the UI and from the recording, and the discarded attempt's thought part is dropped with it.

Non-regression: four shapes that must not change

Each was run through the same real TUI on both trees, and the persisted assistant parts were diffed.

# Shape base head persisted output
B legit JSON object array as the answer rendered rendered identical (12 parts)
C numeric array [1, 2, 3, 5, 8, 13, 21] rendered rendered identical (1 part)
D leading JSON text → real structured tool call text, tool runs, final text, tool runs, final identical after normalising the fixture path
G legit JSON whose string value contains "}</parameter></function>" rendered rendered identical (5 parts), no retry

G is the false-positive path I flagged in round 2 — the quote/escape-aware scan holds in the real stack: one upstream request, no PROTOCOL_TAG_LEAK.

tags inside a JSON string are preserved

D confirms ordering and tool execution survive the buffering path:

json then tool call

The buffering tradeoff, measured

The PR describes the tradeoff qualitatively ("may be delivered at the terminal event rather than incrementally"). I measured it: the mock sends every content chunk, then stalls 12 s before the terminal finish frame, and the harness samples the pane every 500 ms.

Payload base: first pixel head: first pixel
numeric array ([1, 2, …) 555 ms 549 ms — unchanged, still incremental
JSON object array ([{…) 551 ms 12 814 ms — held until the terminal event

At t≈6 s, mid-stall, with all content already sent upstream:

before — text already on screen after — still spinning
before after

Reading of this: the delay is bounded by the terminal event and loses nothing — scenario B's final render and persisted bytes are identical on both trees. It only affects responses whose first non-whitespace character is { or [{, and numeric arrays and Markdown brackets are provably unaffected. For a coding agent that is a rare answer shape, and the cost is perceived latency on a shape that is far more often a leak than an answer. I consider it an acceptable trade; it is worth keeping in mind if someone later reports "my JSON answers feel less streamy".

Checks

  • packages/core/src/core/geminiChat.test.ts on head: 285/285 passed (27.4 s), including the new post-stream-guard case added in 45ed8f111.
  • npm ci + npm run bundle: clean on both trees.
  • Production code is unchanged since my round-4 approval at 4a374ef32; 45ed8f111 adds only a test.

Verdict

Good to merge. The reported production failure reproduces on base and is fixed on head; the fix does not alter any of the four adjacent shapes I could construct; the one behavioural cost is a bounded, lossless streaming delay on a narrow input shape, now quantified above.

中文说明

45ed8f111 上的本地真实环境验证 —— 合并参考

macOS(Node v24.18.1)全链路运行,产品内部不打任何 mock。基于 GitHub merge ref 拉出两个全新的 detached worktree,各自 npm ci + 完整 npm run bundle,以真实交互式 TUI 启动(tmux 120×40 中运行 node dist/cli.js --yolo$HOME/$QWEN_HOME/$TMPDIR 全隔离,通过 OPENAI_* 鉴权)。唯一的替身在上游:一个本地 OpenAI 兼容 SSE 服务,回放 #8207 的真实响应形态并记录每一次请求。

commit 是否包含 hasLeakedToolCallTags
修复前 184365390(merge base)
修复后 26a99e433 = 45ed8f111 合入 184365390 是(dist/chunks/chunk-3YWUEUAM.js

两棵树相差本 PR:git diff --stat 184365390 26a99e433geminiChat.ts + geminiChat.test.ts,842 增 / 34 删。

结论:生产 bug 在 base 复现,在 head 已修复

修复前 —— 泄漏的 JSON 与 </parameter> / </function> 标签被当作助手消息渲染出来,只发出 1 次上游请求,无重试。

修复后 —— 失败轮次不输出任何内容,触发重试,只有成功响应到达 UI。

上游请求日志(同一条 prompt):

base:  1  A attempt=1  A/leaked-json
head:  1  A attempt=1  A/leaked-json
       2  A attempt=2  A/retry-clean     <-- 仅 head 有重试

从磁盘读回的会话持久化(~/.qwen/projects/<slug>/chats/<id>.jsonl):

base  assistant parts: [thought] | "[{\"name\":\"create_node\", ... }]\n\n</parameter>\n</function>\n"
      文件中含 "</parameter>":  true
      文件中含 "create_node":   true

head  assistant parts: "RETRY-OK: dispatching the two subagents through the tool channel."
      文件中含 "</parameter>":  false
      文件中含 "create_node":   false

也就是说,泄漏内容在 UI recording 中都消失了,被丢弃轮次的 thought part 也一并丢掉。

非回归:四种必须保持不变的形态

每种都在两棵树上跑同一套真实 TUI,并对持久化的 assistant parts 做 diff。

# 形态 base head 持久化输出
B 合法 JSON 对象数组作为回答 正常渲染 正常渲染 完全一致(12 parts)
C 数字数组 [1, 2, 3, 5, 8, 13, 21] 正常渲染 正常渲染 完全一致(1 part)
D 先输出 JSON 文本 → 再真实结构化工具调用 文本、工具执行、最终回复 同左 归一化 fixture 路径后完全一致
G 合法 JSON,其字符串值中含 "}</parameter></function>" 正常渲染 正常渲染 完全一致(5 parts),无重试

G 正是我在 round 2 提出的误判路径 —— 引号/转义感知的扫描在真实链路上成立:仅 1 次上游请求,未触发 PROTOCOL_TAG_LEAK

D 证明缓冲路径不破坏顺序,工具也照常执行。

缓冲权衡的实测数据

PR 只定性描述了这个权衡("可能在终止事件时一次性输出,而不是增量输出")。我把它测了出来:mock 发完全部 content chunk 后,在终止 finish 帧前停顿 12 秒,harness 每 500ms 采样一次界面。

Payload base:首次可见 head:首次可见
数字数组([1, 2, … 555 ms 549 ms —— 未变,仍是增量输出
JSON 对象数组([{… 551 ms 12 814 ms —— 被暂存到终止事件

第 6 秒(停顿中途、上游内容已全部发完)的截图对比见上方英文部分。

我的判断:这个延迟以终止事件为上界,且不丢任何内容 —— 场景 B 在两棵树上的最终渲染和持久化字节完全一致。它只影响首个非空白字符为 {[{ 的响应,数字数组和 Markdown 方括号已证明不受影响。对编码 agent 来说这是很少见的回答形态,代价只是这种形态下的感知延迟,而这种形态更多情况下本来就是泄漏而非正常回答。我认为这个取舍可以接受;如果日后有人反馈"JSON 回答的流式感变差了",可以回头看这里。

检查项

  • head 上 packages/core/src/core/geminiChat.test.ts285/285 通过(27.4 s),包含 45ed8f111 新增的 post-stream guard 用例。
  • 两棵树的 npm ci + npm run bundle 均干净通过。
  • 自我 round-4 批准的 4a374ef32 起,生产代码未变;45ed8f111 只加了测试。

结论

可以合并。 上报的生产故障在 base 复现、在 head 修复;该修复未改变我所能构造的四种相邻形态的任何行为;唯一的行为代价是一个有界、无损的流式延迟,作用于很窄的输入形态,且已在上文量化。

@wenshao
wenshao added this pull request to the merge queue Aug 2, 2026
Merged via the queue into QwenLM:main with commit 999587b Aug 2, 2026
61 checks passed
@qwen-code-ci-bot

Copy link
Copy Markdown
Collaborator

Released in v0.21.4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

autofix/takeover Summon the autofix loop to manage this PR (remove to release; needs triage+) review/self-reported The linked issue was opened by the PR author (self-reported)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(core): JSON-style tool call arguments leak as plain text when model drops function-calling format

5 participants