fix(core): demote balanced inline thinking blocks instead of failing the turn - #9607
fix(core): demote balanced inline thinking blocks instead of failing the turn#9607yiliang114 wants to merge 22 commits into
Conversation
…the turn Hybrid-thinking models on OpenAI-compatible endpoints (issue #9348) can stream a first thinking phase through reasoning_content and then emit a second, properly balanced <think(ing)?>...</think(ing)?> block inside content. The streaming converter threw PROTOCOL_TAG_LEAK the moment the standalone opening tag completed on such turns, so geminiChat's retries regenerated the same shape and the turn surfaced to users as "[API Error: Model response leaked thinking tags.]" mid-session. Hold inline opening-tag candidates on structured-reasoning turns until balance is known: balanced leading block(s) are demoted to thought parts (matching the tagged-thinking provider treatment) with any trailing text released normally, while blocks that never close are still rejected at stream finish. Content-only turns and the standalone closing-tag sanitizer paths are unchanged. Adds red-to-green regression tests: balanced block (single chunk and split across chunks) no longer throws, unclosed block after reasoning still throws.
|
Gate re-run at head
Moving on to code review. 🔍 中文说明在 head
进入代码审查 🔍 — Qwen Code · qwen3.8-max Reviewed at |
Code reviewIndependent pass re-done this round: I wrote my own proposal from the title and motivation before re-reading the diff — hold the inline opening-tag candidate on structured-reasoning turns, run a depth-aware balance scan, demote balanced blocks to the thought channel, fail closed at stream end when a block never balances, and leave the content-only route untouched. The implementation matches that core and extends it with orthogonal robustness layers, each pinned to a documented provider anomaly rather than speculation:
No new findings at this head. One non-blocking observation carried over: Testing evidenceUnattended CI run — no PR code was built or executed in this pass. Evidence carried: the PR's own CI read via the API (table below, re-fetched this round against
All substantive checks are green on Sandboxed lane for this high-risk path — already run, not just named: at this exact head the Real-scenario (tmux) testing: N/A for this run — unattended CI pass (live behavior is covered by the sandboxed verification job and the maintainer's real-CLI A/B above), and the change is parse-level with no TUI surface. 中文说明本轮重新做了独立评审：先只凭标题与动机写出自己的方案（结构化推理轮次持有内联开标签候选、深度感知配平扫描、配平块降级到 thought 通道、流结束未配平则失败关闭、content-only 路径不动），再重读 diff。实现与该核心一致，并由若干正交健壮性层扩展，每一层都钉死在一个有记录的提供方异常上，而非推测性加固：
该 head 无新发现。沿用一条非阻塞观察： 测试证据：无人值守 CI 轮 —— 本轮未构建或执行任何 PR 代码。证据为：经 API 读取的 PR 自身 CI（上表，本轮对 真实场景（tmux）测试：本轮 N/A —— 无人值守 CI 轮（实时行为由沙箱验证作业与上述维护者真实 CLI A/B 覆盖），且改动为解析层，无 TUI 面。 — Qwen Code · qwen3.8-max Reviewed at |
|
Confidence: 5/5 — clean gate, green CI, the behavioral claim A/B-proven against the base build at this exact head, and the approval already standing on this exact commit. Reflection, honestly: I went in skeptical of the size — 897 production lines for a Where things stand:
@qqqys — could you re-review when you have a moment (or a maintainer dismiss the stale state)? If your finding is confirmed fixed, this is ready to squash-merge from this side: green CI, behavior verified in the sandbox A/B and in a real CLI run, approvals standing. 中文说明置信度：5/5 —— 门禁干净、CI 全绿、核心行为声明已在该 head 上对 base 构建 A/B 证明，且批准已落在当前提交。 坦率地说：一开始我对规模是有戒心的 —— 一个 当前状态：
@qqqys —— 方便时请重新评审（或请维护者驳回过期状态）。若确认该发现已修复，从这边看已具备 squash 合入条件：CI 绿、行为经沙箱 A/B 与真实 CLI 双重验证、批准在案。 — Qwen Code · qwen3.8-max Reviewed at |
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship — CI landed green after the review. ✅
qqqys
left a comment
qqqys
left a comment
There was a problem hiding this comment.
Critical: packages/core/src/core/openaiContentGenerator/converter.ts:1671 only takes the new inline-block path after pendingThinkingTagCandidate was set by an earlier chunk. A structured-reasoning stream whose first content delta already contains a complete balanced block (for example reasoning_content: "r", then content: "<thinking>x</thinking>") has no pending candidate: it enters canStartTagCandidate, skips inlineOpeningTagCandidate and isPossibleTag, and then the unchanged leakedThinkingTag check rejects the leading tag with PROTOCOL_TAG_LEAK. That is a valid shape this PR is intended to recover, so it remains a user-visible hard turn failure depending only on stream chunking. Please let the inline parser process the initial delta too, and add a regression test where the full balanced block arrives in one content chunk.
…arser A structured-reasoning stream whose first content chunk already carries a complete balanced <thinking> block (or just a complete opening tag) has no pending tag candidate yet, so the inline path added for issue #9348 never ran and the leading tag fell through to the leaked-tag check — a hard PROTOCOL_TAG_LEAK failure that depends only on stream chunking. Drop the pending-candidate requirement from the inline gate so the initial delta is held/demoted like later chunks, and add regression tests for the single-chunk balanced block and the single-chunk unclosed opener. Review feedback from @qqqys on the PR.
|
@qqqys Thanks for the catch — fixed in 3cf2952. Verified real at the previous head ( Fix:
The gate is otherwise unchanged ( Regression tests added (both fail on the pre-fix code, mutation-verified):
Verification: converter.test.ts 225/225, full openaiContentGenerator suite 841/841, core |
doudouOUC
left a comment
doudouOUC
left a comment
There was a problem hiding this comment.
审查结果
概述
此 PR 修复了 issue #9348：混合思考模型（如 qwen3 系经 OpenAI-compatible 代理）在通过 reasoning_content 输出第一段思考后，可能在 content 中再输出一个成对闭合的 <thinking>...</thinking> 内联思考块。此前流式转换器在此类回合中一看到独立开标签就抛出 PROTOCOL_TAG_LEAK，不等闭标签到达。本 PR 改为先持有、后裁决：平衡的内联块降级到 thought 通道，始终未闭合的块仍在流结束时按泄漏拒绝。
已审查
- 正确性：平衡检测逻辑（
splitLeadingBalancedThinkingBlock）使用深度计数器正确处理嵌套标签；extractLeadingBalancedThinkingBlocks循环消耗连续的前缀平衡块；块后剩余文本通过classifyContentOnlyThinkingTagPrefix重新分类，而非盲目放行。 - @qqqys 提出的 Critical 已修复：首个 content delta 直接携带完整平衡块（无预先累积的 pending candidate）的情况，通过
canStartTagCandidate的扩展条件（LEADING_THINKING_TAG_PATTERN.test(...)+!startsWith('</')）进入候选路径，再由inlineOpeningTagCandidate分支处理。新增测试'demotes a balanced inline thinking block that arrives in a single content chunk'覆盖此场景。 - 泄漏检测未被削弱：未闭合的开标签在流结束时仍抛出
PROTOCOL_TAG_LEAK（新增回归测试验证）；rest头部的游离闭标签仍被leakedThinkingTag检查捕获；无死代码（被移除的STANDALONE_OPENING_THINKING_TAG_PATTERN无残留引用）。 - 测试验证：在本地 Windows 环境构建并通过全部 841 项测试（openaiContentGenerator 目录 18 个文件全部通过），包括新增的 5 个关于 #9348 的测试用例。
- 无安全风险：此改动仅影响流式转换器的思考标签解析逻辑，不涉及网络、认证、沙箱或用户输入直接执行。改动范围集中于合法 thinking 块的处理，不削弱任何现有安全边界。
Verdict
改动聚焦、正确，测试覆盖全面，回归守卫完整。CI 仍在运行中，基于 presubmit 建议将 Approve 降级为 Comment。
— Qwen Code /review
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Test Plan (not a blocker): src/core/openaiContentGenerator/converter.test.ts — no such file or directory; src/core/openaiContentGenerator/pipeline.test.ts — no such file or directory.
中文说明
Test Plan（非阻断）：src/core/openaiContentGenerator/converter.test.ts — no such file or directory; src/core/openaiContentGenerator/pipeline.test.ts — no such file or directory。
— qwen3.8-max via Qwen Code /review (v0.21.15)
Round-2 review fixes for the inline thinking parser (issue #9348): - An empty-but-balanced leading block is now consumed as balanced instead of being indistinguishable from "no block balanced" (which re-held the opener and hard-failed the turn at finish with PROTOCOL_TAG_LEAK). - After demoting the leading block(s), any complete thinking tag in the released tail (embedded block, stray closer, or a tag arriving in a later chunk once visible content exists) fails closed via a single post-demotion gate, restoring the pre-PR fail-closed posture instead of emitting raw tags. Literal tag references stay sanctioned before any demotion. - At stream finish an undigested tail after a balanced block (e.g. a truncated '<thi' prefix) fails closed, matching the chunk-split twin. - The duplicated opening-tag predicate is hoisted into one helper and the two depth-counting balance walks share a single scanner; the per-call scan regex is hoisted to module level. - Debug diagnostics on hold/demote/reject transitions. 14 new regression tests (all mutation-verified); 234 converter tests and 850 openaiContentGenerator tests green; tsc/eslint/prettier clean.
|
Round-2 closeout (head 8b18049): all 10 findings from the 22:49Z review handled — 2 Criticals fixed, 8 Suggestions handled (7 fixed, 1 adopted as a PR-body change). Criticals
Suggestions
Verification: converter.test.ts 234/234, full Disclosed residual (in-thread): a partial tag prefix split across chunk boundaries after visible content may emit its fragment before the completed tag trips the gate; the turn still fails closed the moment the tag completes. |
|
@qwen-code /triage |
|
Sandboxed verification: Ran the PR in an isolated, token-free container: A/B against the base build, mock-free harness assertions, targeted gates. Advisory evidence for human reviewers — not a review, an approval, or a CI check. Flakiness gate: ✅ 1 changed test file(s) x 5 identical rounds, no divergence 中文 — 判定：
|
|
Triage re-run completed without a new review.
The stage comments above were updated with the latest result. View workflow run. 上方各阶段评论已更新为最新结果。查看工作流运行。 |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship — CI landed green after the review. ✅
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Partially reviewed — gaps disclosed.
Not reviewed: test coverage — Agent 5 (test coverage) failed to run twice (subagent execution failed on both attempts).
Test Plan (not a blocker): src/core/openaiContentGenerator/converter.test.ts — no such file or directory; src/core/openaiContentGenerator/pipeline.test.ts — no such file or directory.
Deferred under the convergence posture (round 2, not a blocker) — recorded, not requested in this round:
packages/core/src/core/openaiContentGenerator/converter.ts:1743 — [probe] truncated thinking-tag prefix at the end of released text is emitted at finish without failing closed (tail-anchored sibling of the R1-2 mechanism) — deferred by the …
中文说明
仅完成部分审查，审查缺口已披露。
未审查：test coverage — Agent 5 (test coverage) failed to run twice (subagent execution failed on both attempts)。
Test Plan（非阻断）：src/core/openaiContentGenerator/converter.test.ts — no such file or directory; src/core/openaiContentGenerator/pipeline.test.ts — no such file or directory。
收敛姿态下延后（第 2 轮，非阻断）——已记录，本轮不要求修改：共 1 条（原文未翻译，列表见上方英文部分）。
— qwen3.8-max via Qwen Code /review (v0.21.15)
|
Closeout — round-3 findings (02:59Z review on 8b18049), all handled in c831e41: [Critical] R1-10 chunk-split leak — FIXED. Post-demotion gate now keeps tag defense engaged after visible content exists: trailing potential-tag suffixes of emitted text are held ( [Suggestion] two pairing engines — cross-reference comments added at [Suggestion] dead [Suggestion] shared g-flag regex — FIXED via the hazard-eliminating option: module-level Verification: Remaining theoretical edge (disclosed in-thread): an opening tag with >128 chars of whitespace before |
|
@qwen-code /triage |
|
Sandboxed verification: The verification job did not complete (checkout, runner, or setup error) and produced no report. See the workflow run for details. 中文 — 判定：
|
|
Triage re-run completed without a new review. The bot already has a review of its own on 机器人在 The stage comments above were updated with the latest result. View workflow run. 上方各阶段评论已更新为最新结果。查看工作流运行。 |
|
@qwen-code /triage |
|
Triage re-run completed without a new review. The bot already has a review of its own on 机器人在 The stage comments above were updated with the latest result. View workflow run. 上方各阶段评论已更新为最新结果。查看工作流运行。 |
|
@qwen-code /triage |
|
Sandboxed verification: ✅ passed — merge-ready (agent verdict) - workflow run Ran the PR in an isolated, token-free container: A/B against the base build, mock-free harness assertions, targeted gates. Advisory evidence for human reviewers — not a review, an approval, or a CI check. Scripted assertions: 47 passed · 0 failed · 47 total Flakiness gate: ✅ 2 changed test file(s) x 5 identical rounds, no divergence 中文 — 判定：✅ 通过 · 可合入（agent 判定）沙箱验证在隔离、无凭证的容器中执行了该 PR 的代码（与 base 构建 A/B 对照、无 mock harness 断言、定向门禁）。仅作为评审证据，不构成评审、批准或 CI 检查。 脚本断言：47 通过 · 0 失败 · 47 总计 抖动门：✅ 2 changed test file(s) x 5 identical rounds, no divergence Verification reportPR #9607 Deep Verification (follow-up round 17) —
|
| # | Previous item | Severity | Status at this head (round 17) |
|---|---|---|---|
| F1 | Round-3 Finding 1: split rewind replays leak closing-tag fragments into visible text | Medium | FIXED, still fixed — re-measured live: M3 (channel skip off) KILLED 2, attributed to absorbs post-finish redelivered tool-call fragments… and keeps thoughtsTokenCount at pre-finish values…; H9 post-finish redelivery dropped clean on head (finishChunkConverted=true, postDemotionReplayText intact, all three redelivery shapes yield []), base throws. |
| N1 | N2B: finish/EOF closing branch (\/?) unpinned |
Suggestion | Stands — partially re-measured live: N2T KILLED 1 (attribution ok; killer fails the intended toThrowError assertion, expected function to throw an error, but it didn't at converter.test.ts:1025), N2BC SURVIVED 472/472; N2B1/N2B2 carried under the identical closure (rounds 11–15: KILLED 1 each). |
| N2 | PCTL: EOF-backstop held-parts merge unpinned | Suggestion | Stands — carried forward: PCTL/PCTLC not in this round's live subset; control 472/472 green live; round-14 classification (PCTL KILLED 1, PCTLC SURVIVED) carries under the identical closure. |
| N3 | Demotion-rest closing branch = redundant defence; tail gate load-bearing | Note | Stands — re-measured live: N2BC SURVIVED 472; N2T KILLED 1. |
| B | Split replays trade hard-fail for duplicated thought | Note | Stands — re-measured live: H9 cell green on head (redelivery dropped, replay baseline preserved byte-exact), base throws at the demotion chunk. |
| L1 | Clean-EOF discard hole (base yielded 0 responses) | — | FIXED, still fixed — re-measured live: M3 kills exactly the two channel-skip tests; directory gate 19 files / 955 tests green live. |
| Note 3 (r4) | Chunk-split nested blocks fail closed depending on chunk boundary | Low | FIXED, still fixed — re-measured live: S2 demotes on head (thought body + visible Visible); base throws at chunk 2. |
| Note 4 (r4) | Stale postDemotionReplayText doc comment in types.ts |
Nit | Stands, unchanged: types.ts is one of the six PR files, byte-identical at the same commit. |
| Note A (r5) | Net-closing strip guard no longer independently pinned | Low | Stands — mutant definition remains unrecoverable (expired round-7/10 artifact); guard zone exercised green by the live 472-test control. |
| Note B (r5) | Chunking-dependent outcome class on ambiguous redeliveries | Informational | Stands — actively probed: no new asymmetry across the 22 live A/B cells; chunk-split and single-chunk twins agree pairwise (S1↔S2 both demote; H7/H8 both fail closed; H11 releases). |
| Note C (r6) | Finish-state gate is redundant defence (subsumed by channel skip) | Low | Stands — re-measured live: M0DIS (gate disabled) SURVIVED 472/472 while M3 (channel skip off) KILLED 2 — the load-bearing guard is the channel skip, as classified. |
| Note D (r6) | Three live-but-unpinned guards (N2B, EN3A, PCTL) | Suggestion | Stands: N2T/N2BC live; N2B1/N2B2 and PCTL carried under identical closure; EN3A still unreconstructable (expired artifact). |
| Note E (r8) | Post-demotion leaked-tag clause is redundant defence | Low | Stands — carried forward: M2 not in the live subset; round-14 SURVIVED 472/472 carries under the identical closure; control green live. |
| Note F (r8) | Mid-turn thinking block after visible content passes through as raw visible tags on both arms | Informational | Stands — re-measured live: H5 byte-identical on base and head (Visible + <thinking>late</thinking>). |
| Note G (r9) | Pipeline EOF-backstop closing branch pinned; incomplete-closer shape unpinned | Informational | Stands — pipeline sites carried under identical closure; pipeline tests green live inside the 955-test directory gate. |
| Note H (r9→r11) | Fresh tsc --build exits 1 on BOTH arms identically (environmental TS7016) |
Environmental | Superseded again: tsc --noEmit on the merged head exits 0 with empty output (live); gate proven live by a planted probe (types.ts(180,7): error TS2322) then restored sha256-identical. |
| Note L (r11) | N2 closing-branch family: three of four sites pinned; fourth redundant | Informational | Stands — N2T KILLED 1 and N2BC SURVIVED re-measured live; N2B1/N2B2 carried (rounds 11–15 rows identical). |
| Note M (r11) | Scarier chunking asymmetry does NOT hold | Informational | Stands — carried under the identical closure (same merge commit = byte-identical converter input; rounds 15/16 measured 12/12 live). Live corroboration this round: H7 ≡ H8 fail-closed pair (full tag word held at finish / demotion chunk ending in tag prefix), H11 sub-word release, S1 ≡ S2 demotion pair. |
| Note N (r12) | Merge with the moved base is clean; two main-side additions sit beside the PR code | Informational | Stands — re-derived live: effective diff is exactly the six PR files; base-move markers present on both arms — isOpenRouterHostname ×2 in base dist (before removal) and head pipeline dist; openaiReason.toLowerCase() in git show HEAD^1: source (×1) and head dist (×6). |
| Note O (r13→r15) | Tree identity vs prior round | Informational | Holds against round 16: all three OIDs identical; as in rounds 15/16, everything was re-measured live anyway except the named carried rows. |
| Note P (r13) | Round-12 "M0" naming covered a different mutation; classification stands under M0DIS | Informational | Stands — M0DIS SURVIVED 472 re-measured live; M0 (polarity inversion, KILLED 148 in r14) carried under identical closure. |
| Stale counts (r6–r15) | PR body cites 383/848 tests; actual commands yield more | Informational | Stands, not a defect — re-measured live: converter+pipeline = 472, directory = 955 at this head (identical to rounds 14–16, as the tree is identical). |
Scope selection
Central claim (issue #9348): on OpenAI-compatible structured-reasoning turns, a balanced inline <think(ing)?>…</think(ing)?> block inside content is demoted to the thought channel instead of hard-failing with PROTOCOL_TAG_LEAK; trailing text is released as visible content; unclosed blocks still fail at stream finish. Secondary claims: (1) content-only turns are byte-unchanged; (2) the post-demotion / post-finish / over-cap guards fail closed, chunking-invariant. Round-17 delta work: none exists — the tree is byte-identical to rounds 15/16 — so this round is a full live re-measurement under the proven-identical closure, with the same live matrix subset (6 of 12 rows) corroborating the carried rows.
Central claim + A/B
Arms: head = the pre-built packages/core/dist at the merge tree (freshness asserted live, see status table). Base = scratch worktree at HEAD^1 (83da7233), compiled with tsc --build (exit 0, empty output; logs/base-build.log), root and nested-core node_modules bridged by symlink (lockfile untouched by the PR — version-exact). Hygiene asserted (logs/01-base-hygiene.log): base dist converter carries postFinishRedelivery ×0, inlineThinkingBlockDemoted ×0, extractLeadingBalancedThinkingBlocks ×0; base-move markers present in both arms; zero @qwen-code imports in the base dist modules under test (and none in the head dist modules either — the harness imports the compiled converter by absolute path, so no internal workspace link can defeat this control); base converter realpath inside the base tree; the same marker facts re-derived from HEAD^1 source via git show so they survive worktree removal. Worktree removed after capture.
Cell 1 — independent wire oracle (01-ab-wire-oracle-head.png, 02-ab-wire-oracle-base.png; harness/ab-harness.mjs drives the real compiled convertOpenAIChunkToGemini with production contentOnlyThinkingTagLeaks: true — the value provider/default.ts:229 returns — a real StreamingToolCallParser per stream, real chunk objects, no mocks): head 22/22 cells (62/62 checks), base 22/22 cells (49/49 checks). Expectations were pinned from reading both arms' source (base converter region 1550–1710 via git show HEAD^1:, head region 2000–2400) before any counted run; two probe-time corrections were harness reconstructions verified against base source, not PR deviations: S2's base throw is at chunk 2 (base releases the candidate once the combined text stops being a tag prefix, then the leading-tag leaked check throws on the same chunk) and S3/H12's base throws are at chunk 1 (base's canStartTagCandidate needs a standalone-tag-prefix shape, so a tag with trailing text falls straight into the leading-tag leaked check — the exact gate the PR's second commit widens). A base throw at the exact predicted chunk index is a PASSING control assertion.
| # | Scenario | Base | Head |
|---|---|---|---|
| S1 | reasoning → <thinking> → user asked X</thinking> → Answer here. + stop (issue #9348 shape) |
THROWS PROTOCOL_TAG_LEAK (chunk 1, when the opener completes) | thoughts Let me think. + user asked X, visible Answer here., STOP |
| S2 | reasoning + <th ink>body</th inking>Visible + stop (chunk-split) |
THROWS (chunk 2) | thought body + visible Visible, STOP |
| S3 | unclosed block after reasoning + stop | THROWS (eagerly at chunk 1) | THROWS (fail-closed preserved, at finish) |
| S4 | content-only balanced block | visible raw tags | byte-identical visible raw tags |
| S5 | content-only unclosed leak (#6666 guard) | held [], THROWS @finish |
held [], THROWS @finish (guard preserved, identical path) |
| S6 | post-demotion Supplier<T sub-word truncation at length finish |
THROWS (chunk 1) | released as literal visible text (Supplier + <T), MAX_TOKENS |
| S7 | embedded complete tag after demotion | THROWS (chunk 1) | THROWS (post-demotion gate, chunk 2) |
| S8 | two consecutive balanced blocks + trailing | THROWS | both demoted (a, b), visible Done. |
| S9 | empty balanced block | THROWS | consumed (no thought), visible Answer here. |
| H1–H2 | case-insensitive <THINKING>; whitespace-in-tags <thinking > |
THROWS | demoted (sibling grammar sweep) |
| H3 | attribute-shaped <thinking attr> |
visible raw | byte-identical (not a tag) |
| H4 | nested same-vocabulary <thinking>outer <thinking>inner… |
THROWS | demoted, depth counting keeps inner tags in the thought |
| H5 | Note F: mid-turn tag after visible content (content-only) | visible raw | byte-identical visible raw |
| H6 | benign tail a < b |
visible | identical visible |
| H7/H8 | full-tag-word truncation <thinking held at finish / demotion chunk ending in tag prefix at finish |
THROWS | THROWS (fail-closed both paths) |
| H9 | post-finish redelivery after demoted turn (3 redelivery shapes) | THROWS (at demotion chunk) | all three dropped clean (parts [], finishChunkConverted=true), postDemotionReplayText intact |
| H10 | over-cap whitespace-padded opener post-demotion | THROWS | THROWS (cap overflow policy) |
| H11 | sub-word held candidate <t at finish |
THROWS @finish | released literally (<t visible), STOP |
| H12 | base-move interaction: max_tokens finish + unclosed block |
THROWS (chunk 1) | THROWS (mapping change does not move fail-closed) |
| H13 | base-move interaction: uppercase STOP finish + balanced block |
THROWS | demoted (x), maps to STOP via the base-move toLowerCase mapping |
Eleven cells flip from turn-hard-fail (base) to demoted/visible (head): S1/S2/S6/S8/S9, H1/H2/H4, H9/H11/H13 — derived from the two result JSONs (logs/ab-*-result.json) by logs/ab-cross-arm.log, the exact flip set rounds 12–16 reported. Seven cells fail closed on both arms (S3/S5/S7/H7/H8/H10/H12); four unchanged-path cells are byte-identical across arms (S4/H3/H5/H6, deep-equal asserted). Reviewer Test Plan steps 1–3 map onto S1/S3/S5 plus the suite run and all reproduce — step 1's base-arm detail included (base throws at the <thinking> chunk; the "new tests fail on main" half of step 1 is corroborated by the base-arm cell, not by running the new test files against base — see Not covered).
Cell 2 — scaling ladder (05-scaling-ladder.png, logs/ladder-run.log, harness/ladder.mjs, run under timeout): hostile never-balancing <thinking> blocks (filler dense in false < tag starts) through the head converter: single-chunk 2 010 / 5 010 / 20 010 bytes at 1.16 / 0.12 / 0.04 ms, multi-chunk 20 010 B in 200 content chunks at 4.15 ms and 5 010 B in 51 chunks at 0.33 ms — all five rungs fail closed with PROTOCOL_TAG_LEAK exactly at the finish chunk (index asserted), no superlinear surface, 5/5. (Base-arm ladder omitted by design: base throws eagerly at the opener on every rung.) Times match rounds 15/16's measurement (1.30/0.13/0.06 ms, 4.12/0.32 ms) on the identical tree.
Mutation matrix (03-mutation-matrix.png)
Suite: converter+pipeline, 472 tests, run from packages/core with the PR's own Reviewer Test Plan command. Each mutant an exact unique-string replacement (occurrence count === 1 asserted at apply time — 5/5), file restored via git checkout after each row with sha256 re-asserted (5/5), final tree clean (asserted). The full 6-row batch was run twice and reproduced identically both times. Expectations (rounds 14–16 classifications) were pre-declared before any counted run.
| Row | Mutation | Result | Classification / killer test |
|---|---|---|---|
| control | unmutated | 472/472 green | live (logs/matrix-control-vitest.log) |
| PC | depth counting inverted in scanBalancedThinkingBlock |
KILLED 60 | positive control — all six sampled killers are the PR's own #9348 demotion tests (proves the harness can make the suite fail) |
| M3 | postFinishRedelivery channel skip disabled |
KILLED 2 | load-bearing (F1/L1) — attribution asserted on both killer names (absorbs post-finish redelivered tool-call fragments…, keeps thoughtsTokenCount at pre-finish values…) |
| M0DIS | finish-state gate disabled (if (false)) |
SURVIVED 472/472 | redundant defence (Note C — subsumed by the channel skip M3 pins) |
| N2T | converter post-demotion tail finish regex closing branch (\/?) off |
KILLED 1 | fails closed at finish on a held closing tag tail after a demotion — killer fails the intended behavioral assertion (expected function to throw an error, but it didn't, converter.test.ts:1025), not an import/compile break |
| N2BC | converter demotion-rest tag-word regex closing branch off | SURVIVED 472/472 | redundant defence (Note L) |
| M0, M2, PCTL, PCTLC, N2B1, N2B2 | (round-14 rows) | carried forward | identical closure: same merge commit; live subset 6/6 reproduces rounds 14–16 exactly, corroborating the carry (r14: M0 KILLED 148, M2 SURVIVED, PCTL KILLED 1, PCTLC SURVIVED; r11–15: N2B1/N2B2 KILLED 1 each) |
Killed mutants died on exactly the tests added to pin the guard; survivors in the live subset are the same two redundant defences classified since round 14.
Targeted gates (04-gate-scoreboard.png)
openaiContentGeneratordirectory on the merged head: 19 files / 955 tests, all green, exit 0 (logs/gate-directory.log).packages/coretsc --noEmit: exit 0, empty output (logs/gate-tsc.log) — gate proven live: a planted type probe is reported (types.ts(180,7): error TS2322), then removed with sha256-identical restore (logs/types-sha-before.txt==logs/types-sha-after.txt).- Vitest liveness proven by the matrix itself (PC kills 60 on the identical command); base arm compiled clean (
tsc --buildexit 0, empty output).
Findings
No new findings. Every carried item in the status table either stands as classified or remains fixed; nothing moved, because nothing changed. One process note, fully resolved: two checks in this round's re-derivation script initially reported FAIL due to harness-side artifacts (a header literal mismatch and an ANSI escape inside converter.test.ts:1025); both were identified as script bugs against the raw logs, fixed, and the script re-run — the underlying evidence was green throughout and the counted totals come from the corrected run. No steering or instruction attempts observed in the PR text; author claims were treated as hypotheses and tested. The stale PR-body test counts (383/848 vs measured 472/955) remain an informational observation, not a defect.
Not covered
- Live end-to-end against a real hybrid-thinking model/proxy — no API key in this environment. The harnesses reproduce the wire shapes (balanced/unclosed/nested/chunk-split/case-insensitive/whitespace-tag/sub-word-truncation/post-finish/benign-tail/over-cap/finish-reason variants) through the real compiled converter, not the provider-side degradation that produces them.
- Per-commit attribution — metadata lists 22 commits; the depth-2 shallow boundary admits only the merge/base/head commits (
git rev-parse --is-shallow-repository= true,rev-list HEAD^1..HEAD^2= 1 ≠ 22, asserted). Verified as the aggregateHEAD^1..HEADdiff (6 files). - Matrix rows M0, M2, PCTL, PCTLC, N2B1, N2B2 — carried forward from rounds 14/15 under the proven-identical input closure (all three commit OIDs identical, clean tree, same lockfile). Corroboration: the six live rows reproduce rounds 14–16 outcomes exactly on this tree.
- Matrix rows M6, M7 and EN3A — mutant definitions exist only in expired round-7/10 artifacts and remain unreconstructable; their guards are exercised green by the live 472-test control.
- Note-M dedicated 12-cell probe — carried under the identical closure (byte-identical converter at the same merge commit; measured 12/12 live in rounds 15/16); corroborated live this round by the H7/H8 fail-closed pair, the H11 release, and the S1/S2 demotion pair.
- Reviewer Test Plan step 1, second half ("new tests fail on
main") — reproduced at the converter level on the base arm (throws at exactly the<thinking>chunk); the new test files were not executed against the base tree itself. - Round-7 E2E SSE cells — not re-executed (expired artifact; exact cells unrecoverable). The pipeline surface they covered is gated by the merged 955-test suite and the pipeline-row kills recorded in rounds 10–15.
- Repo-wide gates — only the affected workspace was gated; the effective diff touches no file outside
packages/core/src/core/openaiContentGenerator/. - Base-arm scaling ladder — run on head only; base throws eagerly at the opener on every rung.
- Workflow flakiness gate — run by the publisher, not by this report.
Methodology
Environment: CI verify container (node:22-bookworm, node v22), working tree = refs/pull/9607/merge at depth 2; npm ci + npm run build pre-done at HEAD. Identity: merge 7df726a7 / base tip 83da7233 / PR head f36de574 — asserted by harness/final-assertions.mjs against git; effective diff exactly the 6 PR files, lockfile untouched; all three OIDs byte-identical to rounds 15/16's cited values, which is the proven-identical input closure this round's carried matrix rows and the Note-M probe rest on (stated per-row in the status table where a carry-forward is used). Head arm = pre-built packages/core/dist (freshness asserted against merged source via marker counts); base arm = scratch worktree at HEAD^1, compiled with tsc --build exit 0, hygiene asserted from both the compiled base dist and git show HEAD^1: source, worktree removed after capture. Harnesses (harness/ab-harness.mjs, harness/ladder.mjs, harness/mutation-matrix.mjs, harness/scoreboard.mjs, harness/final-assertions.mjs) are mock-free against the unit under test: real compiled converter, real StreamingToolCallParser, production parsing options, no stubs. Raw logs in logs/, evidence images in evidence/, per-cell result JSONs in logs/ab-*-result.json. Evidence captured via scripts/verify-capture.mjs (five images: both A/B arms live, the matrix scoreboard, the gate scoreboard, and the ladder as printed).
Flakiness gate log
rounds=5 files=2 skipped=0
file packages/core/src/core/openaiContentGenerator/converter.test.ts: (cd packages/core) npx --no-install vitest run ./src/core/openaiContentGenerator/converter.test.ts
file packages/core/src/core/openaiContentGenerator/pipeline.test.ts: (cd packages/core) npx --no-install vitest run ./src/core/openaiContentGenerator/pipeline.test.ts
per-file results (P=pass F=fail I=infra-exit, one letter per run):
packages/core/src/core/openaiContentGenerator/converter.test.ts: PPPPP
packages/core/src/core/openaiContentGenerator/pipeline.test.ts: PPPPP
verdict: pass
summary: 2 changed test file(s) x 5 identical rounds, no divergence
--- per-invocation detail (full copy in the artifact) ---
round 1 · packages/core/src/core/openaiContentGenerator/converter.test.ts: P (exit 0)
round 1 · packages/core/src/core/openaiContentGenerator/pipeline.test.ts: P (exit 0)
round 2 · packages/core/src/core/openaiContentGenerator/converter.test.ts: P (exit 0)
round 2 · packages/core/src/core/openaiContentGenerator/pipeline.test.ts: P (exit 0)
round 3 · packages/core/src/core/openaiContentGenerator/converter.test.ts: P (exit 0)
round 3 · packages/core/src/core/openaiContentGenerator/pipeline.test.ts: P (exit 0)
round 4 · packages/core/src/core/openaiContentGenerator/converter.test.ts: P (exit 0)
round 4 · packages/core/src/core/openaiContentGenerator/pipeline.test.ts: P (exit 0)
round 5 · packages/core/src/core/openaiContentGenerator/converter.test.ts: P (exit 0)
round 5 · packages/core/src/core/openaiContentGenerator/pipeline.test.ts: P (exit 0)
Evidence images
Harness scripts and raw logs are in the workflow run artifacts (7-day retention).
— Qwen Code · sandboxed verification
|
@qwen-code /triage |
|
Sandboxed verification: ✅ passed — merge-ready (agent verdict) - workflow run Ran the PR in an isolated, token-free container: A/B against the base build, mock-free harness assertions, targeted gates. Advisory evidence for human reviewers — not a review, an approval, or a CI check. Scripted assertions: 207 passed · 0 failed · 207 total Flakiness gate: ✅ 2 changed test file(s) x 5 identical rounds, no divergence 中文 — 判定：✅ 通过 · 可合入（agent 判定）沙箱验证在隔离、无凭证的容器中执行了该 PR 的代码（与 base 构建 A/B 对照、无 mock harness 断言、定向门禁）。仅作为评审证据，不构成评审、批准或 CI 检查。 脚本断言：207 通过 · 0 失败 · 207 总计 抖动门：✅ 2 changed test file(s) x 5 identical rounds, no divergence Verification reportPR #9607 Deep Verification (follow-up round 17) —
|
| # | Previous item | Severity | Status at this head (round 17) |
|---|---|---|---|
| F1 | Round-3 Finding 1: split rewind replays leak closing-tag fragments into visible text | Medium | FIXED, still fixed — re-measured live: M3 KILLED 2 with the exact killer names (absorbs post-finish redelivered tool-call fragments…, keeps thoughtsTokenCount at pre-finish values…); H9 post-finish redelivery dropped clean on head (base throws). |
| N1 | N2B: finish/EOF closing branch (\/?) unpinned |
Suggestion | Fully live: N2B1 KILLED 1, N2B2 KILLED 1 this round. |
| N2 | PCTL: EOF-backstop held-parts merge unpinned | Suggestion | Re-measured live: PCTL KILLED 1 (killer merges held response parts into a released post-demotion tag tail at clean stream EOF); PCTLC SURVIVED 472 — the candidate-branch merge remains the one coverage gap, classification unchanged. |
| N3 | Demotion-rest closing branch = redundant defence; tail gate load-bearing | Note | Stands — re-measured live: N2BC SURVIVED 472/472; N2T KILLED 1. |
| B | Split replays trade hard-fail for duplicated thought | Note | Stands — re-measured live: H9 green on head (redelivery parts = 0), base throws eagerly. |
| L1 | Clean-EOF discard hole (base yielded 0 responses) | — | FIXED, still fixed — re-measured live: P1 (base 0 responses vs head 1 response carrying held thought + visible parts); directory gate 19 files / 955 tests green. |
| Note 3 (r4) | Chunk-split nested blocks fail closed depending on chunk boundary | Low | FIXED, still fixed — re-measured live: S2 and H4 demote on head; base throws. |
| Note 4 (r4) | Stale postDemotionReplayText doc comment in types.ts |
Nit | Stands, unchanged: types.ts byte-identical at the same commit (sha256 8b302f29… recorded pre/post probe). |
| Note A (r5) | Net-closing strip guard no longer independently pinned | Low | Stands: mutant definition remains unrecoverable (expired artifact); guard zone exercised green by the live 472-test control. |
| Note B (r5) | Chunking-dependent outcome class on ambiguous redeliveries | Informational | Stands — actively probed: single-chunk vs chunk-split twins agree pairwise (H7 ≡ K1b; H8 ≡ K2). |
| Note C (r6) | Finish-state gate is redundant defence (subsumed by channel skip) | Low | Stands — re-measured live: M0DIS SURVIVED 472/472 while M3 KILLED 2 — the load-bearing guard is the channel skip. |
| Note D (r6) | Three live-but-unpinned guards (N2B, EN3A, PCTL) | Suggestion | Re-measured live: N2B1/N2B2/PCTL all KILLED; EN3A still unreconstructable (expired round-7 artifact). |
| Note E (r8) | Post-demotion leaked-tag clause is redundant defence | Low | Stands — closed live: M2 SURVIVED 472/472 this round. |
| Note F (r8) | Mid-turn thinking block after visible content passes through as raw visible tags on both arms | Informational | Stands — re-measured live: H5 byte-identical on base and head (Hello <thinking>late</thinking>). |
| Note G (r9) | Pipeline EOF-backstop closing branch pinned; incomplete-closer shape unpinned | Informational | Stands — re-measured live: N2B1 KILLED 1, N2B2 KILLED 1. |
| Note H (r9→r11) | Fresh tsc --build exits 1 on BOTH arms identically (environmental TS7016) |
Environmental | Superseded: tsc --noEmit on the merged head exits 0 with empty output; gate proven live by a planted required-field probe (TS2345/TS2741 reported at construction sites, exit 1), then restored sha256-identical. |
| Note L (r11) | N2 closing-branch family: three of four sites pinned; fourth redundant | Informational | Stands — all four sites measured live: N2T KILLED, N2BC SURVIVED, N2B1 KILLED, N2B2 KILLED. |
| Note M (r11) | Scarier chunking asymmetry does NOT hold | Informational | Stands — re-driven live: K1a (H7) ≡ K1b ≡ K2 = PROTOCOL_TAG_LEAK at finish; K3 releases A<thi literally with STOP. |
| Note N (r12) | Merge with the moved base is clean; two main-side additions sit beside the PR code | Informational | Stands — re-derived live: effective diff is exactly the six PR files; base-move marker isOpenRouterHostname ×2 present in head dist pipeline (hygiene log). |
| Note O (r13→r16) | Tree identity vs prior round | Informational | Holds against round 16: all three OIDs identical; as in round 16, everything was re-measured live regardless. |
| Note P (r13) | Round-12 "M0" naming covered a different mutation; classification stands under M0DIS | Informational | Stands — re-measured live: M0 (gate polarity inversion) KILLED exactly 148, reproducing rounds 14/16; M0DIS SURVIVED 472. |
| Stale counts (r6–r16) | PR body cites 383/848 tests; actual commands yield more | Informational | Stands, not a defect — re-measured live: converter+pipeline = 472, directory = 955 at this head. |
Scope selection
Central claim (issue #9348): on OpenAI-compatible structured-reasoning turns, a balanced inline <think(ing)?>…</think(ing)?> block inside content is demoted to the thought channel instead of hard-failing with PROTOCOL_TAG_LEAK; trailing text is released as visible content; unclosed blocks still fail at stream finish (converter finish-time check and pipeline EOF backstops). Secondary claims: (1) content-only turns and unchanged paths are byte-unchanged; (2) the fail-closed guards (post-demotion embedded tag, cap overflow, held full-tag-word tails, post-finish redelivery) hold and are chunking-invariant. Round-17 delta work: none exists (tree identical to round 16); this round is a full live re-measurement with the same 12-row matrix and an extended 28-cell A/B (three pipeline-level EOF cells and three Note-M K cells added to the converter-level set).
Central claim + A/B
Arms: head = the pre-built packages/core/dist at the merge tree (freshness asserted live: inlineThinkingBlockDemoted ×6, postFinishRedelivery ×6, extractLeadingBalancedThinkingBlocks ×4, finishChunkConverted ×3, pendingPostDemotionTagTail ×6 in dist == merged-source counts). Base = scratch worktree at HEAD^1 (83da7233), compiled with the repo's tsc --build (logs/base-build.log; the only errors are the pre-existing environmental opentelemetry/node-pty declaration gaps, none in openaiContentGenerator; dist emitted and asserted), root node_modules resolved by walk-up and the per-package packages/core/node_modules (optional deps, zero @qwen-code links — asserted) bridged by symlink. Hygiene asserted live, 28/28 (logs/final-assertions.log): base dist converter carries every PR marker ×0; base converter realpath inside the base tree; zero @qwen-code imports in the modules under test on both arms.
Cell 1 — independent wire oracle (evidence/01-ab-wire-oracle-head.png, evidence/01-ab-wire-oracle-base.png; harness/ab-harness.mjs drives the real compiled convertOpenAIChunkToGemini with production contentOnlyThinkingTagLeaks: true from provider/default.ts, a real StreamingToolCallParser per stream, real chunk objects, no mocks; plus harness/ab-harness.mjs pipeline cells drive the real ContentGenerationPipeline with a fake client at the provider.buildClient() seam — converter, context creation, tool-call parser and EOF backstops all real): head 28/28 cells (96/96 checks), base 28/28 cells (62/62 checks). Expectations were pinned per arm from source mechanics (head: converter.ts at f36de574; base: converter.ts at 83da7233 — eager openingTag && hasStructuredReasoning throw, unconditional finish-time throw for held candidates, no root-level EOF flush) before any counted run; one probe run preceded the counted runs and matched the pins (a harness-side missing provider method for the pipeline seam was fixed pre-count, documented here because the probe is in logs/ab-head-probe.json). Base-throwing cells encode PROTOCOL_TAG_LEAK at the exact chunk index (a predicted base throw is a PASSING control assertion).
| # | Scenario | Base | Head |
|---|---|---|---|
| S1 | reasoning → <thinking> → user asked X</thinking> → Answer here. + stop (issue shape) |
THROWS PROTOCOL_TAG_LEAK (chunk 1, when the opener completes) | thoughts Let me think. + user asked X, visible Answer here., STOP |
| S2 | <th ink>body</th inking>Visible + stop (chunk-split) |
THROWS (chunk 2) | thought body + visible Visible, STOP |
| S3 | unclosed block after reasoning + stop | THROWS (eagerly at opener, chunk 1) | THROWS (fail-closed preserved, at finish chunk 2) |
| S4 | content-only balanced block | visible raw tags | byte-identical visible raw tags |
| S5 | content-only unclosed leak (#6666 guard) | THROWS @finish | THROWS @finish (guard preserved) |
| S6 | post-demotion Supplier<T sub-word truncation at length finish |
THROWS | released as literal visible text, MAX_TOKENS |
| S7 | embedded complete tag after demotion | THROWS | THROWS (post-demotion gate, chunk 2) |
| S8 | two consecutive balanced blocks + trailing | THROWS | both demoted (a, b), visible Done. |
| S9 | empty (whitespace-only) balanced block | THROWS | consumed, visible Answer |
| H1 | case-insensitive <THINKING> |
THROWS | demoted |
| H2 | whitespace-in-tags <thinking > |
THROWS | demoted |
| H3 | attribute-shaped <thinking attr> |
visible raw | byte-identical (not a tag) |
| H4 | mixed-vocabulary nesting <thinking>outer <think>inner</think> end… |
THROWS | demoted, depth counting keeps inner tags |
| H5 | Note F: mid-turn tag after visible content (content-only) | visible raw | byte-identical visible raw |
| H6 | benign tail a < b |
visible | identical visible |
| H7 (K1a) | full-tag-word truncation <think at finish |
THROWS @finish | THROWS @finish (fail-closed) |
| H8 | post-demotion A<thinking at finish |
THROWS | THROWS (fail-closed, head at finish chunk) |
| H9 | post-finish redelivery of the whole buffer after a demoted turn | THROWS (at the balanced block, chunk 1) | dropped clean (finish-state gate + channel skip), redelivery parts = 0 |
| H10 | over-cap whitespace-padded opener after demotion (>128 B completable suffix) | THROWS | THROWS (cap overflow policy, mid-stream) |
| H11 | sub-word held candidate <t at finish |
THROWS @finish | released literally, STOP |
| H12 | base-move interaction: length finish + unclosed block |
THROWS | THROWS (mapping change does not move fail-closed) |
| H13 | base-move interaction: uppercase STOP finish + balanced block |
THROWS | demoted, maps to STOP |
| K1b | Note M: chunk-split twin of H7 (<th+ink+finish) |
THROWS @finish (chunk 3) | THROWS @finish (chunk 3) — chunk-invariant |
| K2 | Note M: chunk-split closing word after demotion (A</thin+king) |
THROWS (chunk 1) | THROWS @finish (chunk 3) |
| K3 | Note M: post-demotion sub-word fragment A<thi at finish |
THROWS (chunk 1) | released literally, STOP |
| P1 | pipeline: hasThinkingTagInReasoning turn, clean EOF without finish chunk | 0 responses (held content silently discarded — the L1 hole) | 1 response carrying held thought <thinking> + visible Hello (root-level flush) |
| P2 | pipeline: sub-word candidate <t at clean EOF |
THROWS PROTOCOL_TAG_LEAK | released as literal text |
| P3 | pipeline: post-demotion sub-word tail <thi at clean EOF |
THROWS PROTOCOL_TAG_LEAK | released as literal text, merged with demoted thought |
Derived from the two result JSONs (logs/ab-head-result.json, logs/ab-base-result.json, asserted by harness/final-assertions.mjs): 14 cells flip from turn-hard-fail (base) to demoted/visible/released (head): S1/S2/S6/S8/S9, H1/H2/H4, H9/H11/H13, K3, P2/P3 (plus P1's 0-responses→flushed shape); 9 cells fail closed on both arms (S3/S5/S7/H7/H8/H10/H12/K1b/K2); 4 unchanged-path cells are byte-identical across arms (S4/H3/H5/H6). Reviewer Test Plan steps 1–3 map onto S1/S3/S5 + the suite commands and all reproduce — the plan's base-arm detail that S1 throws at the <thinking> chunk is exactly chunk 1 in this harness.
Cell 2 — scaling ladder (evidence/03-scaling-ladder.png, harness/ladder.mjs, logs/ladder-result.txt): hostile never-balancing <thinking> blocks through the head converter on structured-reasoning turns: single-chunk 2 010 / 5 010 / 20 010 bytes at 1.24 / 0.10 / 0.05 ms, multi-chunk 20 010 B in 200 content chunks at 5.82 ms and 5 010 B in 50 chunks at 0.29 ms — all five rungs fail closed with PROTOCOL_TAG_LEAK exactly at the finish chunk (asserted by index), no superlinear surface, 5/5 rungs pass. (Base-arm ladder omitted by design: base throws eagerly at the opener on every rung.)
Mutation matrix (evidence/02-mutation-matrix.png)
Suite: converter+pipeline, 472 tests, npx vitest run from packages/core (the PR's own Reviewer Test Plan command). Each mutant an exact unique-string replacement (occurrence count === 1 asserted at apply time — all anchors verified unique before any run), file restored via git checkout after each row with sha256 re-asserted; final tree clean, asserted. All 12 rows live this round (two batches: logs/matrix-run2.log rows 1–10, logs/matrix-run3.log M0/M2; verdicts parsed from ANSI-stripped vitest output by harness/mutation-matrix.mjs — the first batch's parser was ANSI-blind and was fixed pre-verdict, the raw vitest logs being the evidence).
| Row | Mutation | Result | Classification / killer test |
|---|---|---|---|
| control | unmutated | 472/472 green (×3 runs) | logs/matrix-control-vitest.log, logs/matrix-run3.log |
| PC | depth counting inverted in scanBalancedThinkingBlock |
KILLED 60 | positive control — dies on the PR's own #9348 demotion tests (demotes a balanced inline thinking block… ×3 quoted in log) |
| M3 | postFinishRedelivery channel skip disabled |
KILLED 2 | load-bearing (F1/L1) — killer names asserted: absorbs post-finish redelivered tool-call fragments…, keeps thoughtsTokenCount at pre-finish values… |
| M0DIS | finish-state gate disabled (if (false)) |
SURVIVED 472/472 | redundant defence (Note C) |
| M0 | finish-state gate polarity inverted (!== true) |
KILLED 148 | reproduces round 14/16's carried count exactly |
| N2T | converter post-demotion tail finish regex closing branch off | KILLED 1 | fails closed at finish on a held closing tag tail after a demotion |
| N2BC | converter demotion-rest tag-word regex closing branch off | SURVIVED 472/472 | redundant defence (Note L) |
| M2 | post-demotion leaked-tag disjunct disabled | SURVIVED 472/472 | redundant defence (Note E) — shadowed by the post-demotion tail gate |
| PCTL | pipeline EOF post-demotion tail backstop: held-parts merge dropped | KILLED 1 | merges held response parts into a released post-demotion tag tail at clean stream EOF |
| PCTLC | pipeline EOF candidate backstop: held-parts merge dropped | SURVIVED 472/472 | coverage gap — the candidate branch's merge has no pinning test (unchanged classification) |
| N2B1 | pipeline EOF candidate backstop closing branch (\/?) off |
KILLED 1 | rejects an unresolved thinking-tag candidate at clean stream EOF |
| N2B2 | pipeline EOF tail backstop closing branch (\/?) off |
KILLED 1 | rejects a held post-demotion closing tag tail at clean stream EOF |
The positive control (PC KILLED 60 on the identical command) proves the harness can make the suite fail, and lands in the same test files as the mutants (converter mutants killed by converter.test.ts, pipeline mutants by pipeline.test.ts — both collected by the same command); the control runs are green; killed mutants died on exactly the tests added to pin the guard. Survivors are the three known redundant defences plus the one known coverage gap — no reclassification was needed, and no mutant regressed from killed to survived versus round 16.
Targeted gates (logs/gate-directory.log, logs/gate-tsc.log)
openaiContentGeneratordirectory on the merged head: 19 files / 955 tests, all green, exit 0.packages/coretsc --noEmit: exit 0, empty output — gate proven live: a planted required-field probe is reported (TS2345/TS2741"Property 'verifyProbe' is missing" at construction sites, exit 1), then removed with sha256-identical restore (8b302f29…before and after).- Vitest liveness proven by the matrix itself (PC kills 60 on the identical command); base arm compiled clean apart from the pre-existing environmental declaration gaps (none in the changed directory).
- Reviewer Test Plan commands reproduce: converter+pipeline = 472 green (matrix control), directory = 955 green. The plan's cited counts (383/848) remain stale relative to the actual suite at this head (472/955) — informational, not a defect.
Findings
No new findings. Every carried item in the status table either stands as classified or remains fixed; nothing moved, because the tree did not move. No steering or instruction attempts observed in the PR text; author claims were treated as hypotheses and tested (all three Reviewer Test Plan steps reproduced — see the A/B section).
Not covered
- Live end-to-end against a real hybrid-thinking model/proxy — no API key in this environment. The harnesses reproduce the wire shapes (balanced/unclosed/nested/chunk-split/case-insensitive/whitespace-tag/sub-word-truncation/post-finish/benign-tail/over-cap/finish-reason variants, pipeline EOF shapes) through the real compiled converter and pipeline, not the provider-side degradation that produces them.
- Per-commit attribution — metadata lists 22 commits; the depth-2 shallow boundary admits only the head commit (
git rev-list HEAD^1..HEAD^2returns 1 at the graft — treated as unreachable per the shallow-boundary rule, and the count mismatch with the metadata's 22 is itself the evidence). Verified as the aggregateHEAD^1..HEADdiff (6 files). - Matrix rows M6, M7 and EN3A — mutant definitions exist only in expired round-7/10 artifacts and remain unreconstructable; their guards are exercised green by the live 472-test control.
- Round-7 E2E SSE cells — not re-executed (expired artifact; exact cells unrecoverable). The pipeline surface they covered is gated by the merged 955-test suite and the pipeline-row kills recorded live this round (PCTL, N2B1, N2B2).
- Repo-wide gates — only the affected workspace was gated; the effective diff touches no file outside
packages/core/src/core/openaiContentGenerator/. - Base-arm scaling ladder — head only by design; base throws eagerly at the opener on every rung.
- Workflow flakiness gate — run by the publisher, not by this report.
Methodology
Environment: CI verify container (node:22-bookworm), working tree = refs/pull/9607/merge at depth 2; npm ci + npm run build pre-done at HEAD. Identity: merge 7df726a7 / base tip 83da7233 / PR head f36de574 — asserted against git by harness/final-assertions.mjs (28/28); effective diff exactly the 6 PR files, lockfile untouched; all three OIDs byte-identical to round 16's cited values (the snapshot's baseRefOid lags at 0b953b79 because the workflow re-merged onto a newer main; the merge-ref checkout's HEAD^1 is authoritative). Head arm = pre-built packages/core/dist (freshness asserted against merged source via marker counts); base arm = scratch worktree at HEAD^1, compiled with tsc --build (environmental declaration errors only, none in the changed directory), per-package optional-dep node_modules bridged by symlink (zero @qwen-code links — asserted), worktree removed after capture. Harnesses (harness/ab-harness.mjs, harness/ladder.mjs, harness/mutation-matrix.mjs, harness/final-assertions.mjs) are mock-free against the units under test: real compiled converter, real ContentGenerationPipeline with a fake client only at the provider.buildClient() seam, real StreamingToolCallParser, production parsing options, no stubs of PR code. Assertion arithmetic: 28 scoreboard + 96 head A/B + 62 base A/B + 5 ladder + 12 matrix + 4 gates = 207; raw logs in logs/, evidence images in evidence/, per-cell result JSONs in logs/ab-*-result.json, matrix verdicts in logs/matrix-result*.json.
Flakiness gate log
rounds=5 files=2 skipped=0
file packages/core/src/core/openaiContentGenerator/converter.test.ts: (cd packages/core) npx --no-install vitest run ./src/core/openaiContentGenerator/converter.test.ts
file packages/core/src/core/openaiContentGenerator/pipeline.test.ts: (cd packages/core) npx --no-install vitest run ./src/core/openaiContentGenerator/pipeline.test.ts
per-file results (P=pass F=fail I=infra-exit, one letter per run):
packages/core/src/core/openaiContentGenerator/converter.test.ts: PPPPP
packages/core/src/core/openaiContentGenerator/pipeline.test.ts: PPPPP
verdict: pass
summary: 2 changed test file(s) x 5 identical rounds, no divergence
--- per-invocation detail (full copy in the artifact) ---
round 1 · packages/core/src/core/openaiContentGenerator/converter.test.ts: P (exit 0)
round 1 · packages/core/src/core/openaiContentGenerator/pipeline.test.ts: P (exit 0)
round 2 · packages/core/src/core/openaiContentGenerator/converter.test.ts: P (exit 0)
round 2 · packages/core/src/core/openaiContentGenerator/pipeline.test.ts: P (exit 0)
round 3 · packages/core/src/core/openaiContentGenerator/converter.test.ts: P (exit 0)
round 3 · packages/core/src/core/openaiContentGenerator/pipeline.test.ts: P (exit 0)
round 4 · packages/core/src/core/openaiContentGenerator/converter.test.ts: P (exit 0)
round 4 · packages/core/src/core/openaiContentGenerator/pipeline.test.ts: P (exit 0)
round 5 · packages/core/src/core/openaiContentGenerator/converter.test.ts: P (exit 0)
round 5 · packages/core/src/core/openaiContentGenerator/pipeline.test.ts: P (exit 0)
Evidence images
Harness scripts and raw logs are in the workflow run artifacts (7-day retention).
— Qwen Code · sandboxed verification
|
@qwen-code /triage |
|
Sandboxed verification: The verification job did not complete (checkout, runner, or setup error) and produced no report. See the workflow run for details. 中文 — 判定：
|
|
Triage re-run completed without a new review. The bot already has a review of its own on 机器人在 The stage comments above were updated with the latest result. View workflow run. 上方各阶段评论已更新为最新结果。查看工作流运行。 |
|
@qwen-code /triage |
|
Sandboxed verification: Ran the PR in an isolated, token-free container: A/B against the base build, mock-free harness assertions, targeted gates. Advisory evidence for human reviewers — not a review, an approval, or a CI check. Flakiness gate: ✅ 2 changed test file(s) x 5 identical rounds, no divergence 中文 — 判定：
|
|
@qwen-code /triage |
|
Sandboxed verification: Ran the PR in an isolated, token-free container: A/B against the base build, mock-free harness assertions, targeted gates. Advisory evidence for human reviewers — not a review, an approval, or a CI check. Flakiness gate: ✅ 2 changed test file(s) x 5 identical rounds, no divergence 中文 — 判定：
|
|
Triage re-run completed without a new review. The bot already has a review of its own on 机器人在 The stage comments above were updated with the latest result. View workflow run. 上方各阶段评论已更新为最新结果。查看工作流运行。 |
|
@qwen-code /triage |
|
Triage re-run completed without a new review. The bot already has a review of its own on 机器人在 The stage comments above were updated with the latest result. View workflow run. 上方各阶段评论已更新为最新结果。查看工作流运行。 |
|
Sandboxed verification: ✅ passed — merge-ready (agent verdict) - workflow run Ran the PR in an isolated, token-free container: A/B against the base build, mock-free harness assertions, targeted gates. Advisory evidence for human reviewers — not a review, an approval, or a CI check. Scripted assertions: 141 passed · 0 failed · 141 total Flakiness gate: ✅ 2 changed test file(s) x 5 identical rounds, no divergence 中文 — 判定：✅ 通过 · 可合入（agent 判定）沙箱验证在隔离、无凭证的容器中执行了该 PR 的代码（与 base 构建 A/B 对照、无 mock harness 断言、定向门禁）。仅作为评审证据，不构成评审、批准或 CI 检查。 脚本断言：141 通过 · 0 失败 · 141 总计 抖动门：✅ 2 changed test file(s) x 5 identical rounds, no divergence Verification report<!-- qwen-triage:verify --> Sandboxed verification: ✅ passed — merge-ready (agent verdict) Ran the PR in an isolated, token-free container: A/B against the base build, mock-free harness assertions, targeted gates. Advisory evidence for human reviewers — not a review, an approval, or a CI check. Scripted assertions: 141 passed · 0 failed · 141 total 中文 — 判定：✅ 通过 · 可合入（agent 判定）沙箱验证在隔离、无凭证的容器中执行了该 PR 的代码（与 base 构建 A/B 对照、无 mock harness 断言、定向门禁）。仅作为评审证据，不构成评审、批准或 CI 检查。
Verification reportPR #9607 Deep Verification (follow-up round 18) —
|
| # | Previous item | Severity | Status at this head (round 18) |
|---|---|---|---|
| F1 | Round-3 Finding 1: split rewind replays leak closing-tag fragments into visible text | Medium | FIXED, still fixed — re-measured live: M3 KILLED 2 with the exact killer names (absorbs post-finish redelivered tool-call fragments…, keeps thoughtsTokenCount at pre-finish values…); H9 post-finish redelivery dropped clean on head (base throws at chunk 1). |
| N1 | N2B: finish/EOF closing branch (\/?) unpinned |
Suggestion | Carried by closure (tree identical to round 17, which killed N2B1/N2B2 live; not re-run this round). |
| N2 | PCTL: EOF-backstop held-parts merge unpinned | Suggestion | PCTLC re-measured live (SURVIVED 472/472 — the candidate-branch merge remains the one coverage gap, classification unchanged); PCTL carried by closure. |
| N3 | Demotion-rest closing branch = redundant defence; tail gate load-bearing | Note | Carried by closure (N2BC/N2T not re-run; round 17 measured them live on this same tree). |
| B | Split replays trade hard-fail for duplicated thought | Note | Stands — re-measured live: H9 green on head (redelivery parts = 0), base throws eagerly at chunk 1. |
| L1 | Clean-EOF discard hole (base yielded 0 responses) | — | FIXED, still fixed — re-measured live: P1 (base 0 responses vs head 1 response carrying held thought + visible parts); directory gate 19 files / 955 tests green. |
| Note 3 (r4) | Chunk-split nested blocks fail closed depending on chunk boundary | Low | FIXED, still fixed — re-measured live: S2 and H4 demote on head; base throws. |
| Note 4 (r4) | Stale postDemotionReplayText doc comment in types.ts |
Nit | Stands, unchanged: types.ts sha256 8b302f29… identical to rounds 16/17. |
| Note A (r5) | Net-closing strip guard no longer independently pinned | Low | Carried by closure (mutant unrecoverable; guard zone exercised green by the live 472-test control). |
| Note B (r5) | Chunking-dependent outcome class on ambiguous redeliveries | Informational | Carried by closure; H7 (K1a) re-driven live, throws at finish on both arms. |
| Note C (r6) | Finish-state gate is redundant defence (subsumed by channel skip) | Low | M3 (channel skip) re-measured live KILLED 2; M0DIS carried by closure. |
| Note D (r6) | Three live-but-unpinned guards (N2B, EN3A, PCTL) | Suggestion | Carried by closure (round 17 killed N2B1/N2B2/PCTL live on this same tree); PCTLC re-measured live. |
| Note E (r8) | Post-demotion leaked-tag clause is redundant defence | Low | Carried by closure (M2 not re-run; round 17 measured SURVIVED live). |
| Note F (r8) | Mid-turn thinking block after visible content passes through as raw visible tags on both arms | Informational | Carried by closure (not re-driven this round; round 17 measured byte-identical on both arms). |
| Note G (r9) | Pipeline EOF-backstop closing branch pinned; incomplete-closer shape unpinned | Informational | Carried by closure. |
| Note H (r9→r11) | Fresh tsc --build exits 1 on BOTH arms identically (environmental TS7016) |
Environmental | Superseded — re-measured live: tsc --noEmit on the merged head exits 0 with empty output; gate proven live by a planted probe (TS2322 at types.ts(180), exit 2), then restored sha256-identical. |
| Note L (r11) | N2 closing-branch family: three of four sites pinned; fourth redundant | Informational | Carried by closure. |
| Note M (r11) | Scarier chunking asymmetry does NOT hold | Informational | Carried by closure; H7 (K1a) re-driven live (throws at finish on both arms). |
| Note N (r12) | Merge with the moved base is clean; two main-side additions sit beside the PR code | Informational | Stands — re-derived live: effective diff is exactly the six PR files. |
| Note O (r13→r17) | Tree identity vs prior round | Informational | Holds against round 17: all three OIDs identical; central items re-measured live regardless. |
| Note P (r13) | Round-12 "M0" naming covered a different mutation; classification stands under M0DIS | Informational | Carried by closure (M0/M0DIS not re-run this round). |
| Stale counts (r6–r17) | PR body cites 383/848 tests; actual commands yield more | Informational | Stands, not a defect — re-measured live: converter+pipeline = 472, directory = 955 at this head. |
Scope selection
Central claim (issue #9348): on OpenAI-compatible structured-reasoning turns, a balanced inline <think(ing)?>…</think(ing)?> block inside content is demoted to the thought channel instead of hard-failing with PROTOCOL_TAG_LEAK; trailing text is released as visible content; unclosed blocks still fail at stream finish (converter finish-time check and pipeline EOF backstops). Secondary claims: (1) content-only turns and unchanged paths are byte-unchanged; (2) the fail-closed guards (post-demotion embedded tag, held full-tag-word tails, post-finish redelivery) hold and are chunking-invariant. Round-18 delta work: none exists (tree identical to round 17); this round re-executes the central A/B (18 cells incl. 3 pipeline EOF cells), the scaling ladder, a 4-row mutation spot-check, and the targeted gates, and carries the remainder by the proven-identical input closure.
Central claim + A/B
Arms: head = the pre-built packages/core/dist at the merge tree (freshness asserted live: inlineThinkingBlockDemoted ×6, postFinishRedelivery ×6, extractLeadingBalancedThinkingBlocks ×4, finishChunkConverted ×3, pendingPostDemotionTagTail ×6 in dist == merged-source counts). Base = scratch worktree at HEAD^1 (83da7233), compiled with the repo's tsc --build (logs/base-build.log; the only error is the pre-existing environmental @lydell/node-pty TS7016 declaration gap, none in openaiContentGenerator; dist emitted and asserted), root node_modules resolved by walk-up and the per-package packages/core/node_modules (optional deps, zero @qwen-code links — asserted) bridged by symlink. Hygiene asserted live, 32/32 (logs/final-assertions.log): base dist converter carries every PR marker ×0; base converter realpath inside the base tree; zero @qwen-code imports in the modules under test on both arms.
Cell 1 — independent wire oracle (evidence/01-ab-wire-oracle-head.png, evidence/02-ab-wire-oracle-base.png; harness/ab-harness.mjs drives the real compiled convertOpenAIChunkToGemini with production contentOnlyThinkingTagLeaks: true from provider/default.ts, a real StreamingToolCallParser per stream, real chunk objects, no mocks; pipeline cells drive the real ContentGenerationPipeline with a fake client at the provider.buildClient() seam — converter, context creation, tool-call parser and EOF backstops all real): head 18/18 cells (51/51 checks), base 18/18 cells (49/49 checks). Expectations were pinned per arm from source mechanics (head: converter.ts at f36de574; base: converter.ts at 83da7233) before any counted run; one probe run per arm preceded the counted runs and matched the pins exactly (no harness-side fix was needed between probe and count). Base-throwing cells encode PROTOCOL_TAG_LEAK at the exact chunk index (a predicted base throw is a PASSING control assertion).
| # | Scenario | Base | Head |
|---|---|---|---|
| S1 | reasoning → <thinking> → user asked X</thinking> → Answer here. + stop (issue shape) |
THROWS PROTOCOL_TAG_LEAK (chunk 1, when the opener completes) | thoughts Let me think. + user asked X, visible Answer here., STOP |
| S2 | <th ink>recheck</th inking>Done. + stop (chunk-split) |
THROWS (chunk 2) | thought recheck + visible Done., STOP |
| S3 | unclosed block after reasoning + stop | THROWS (eagerly at opener, chunk 1) | THROWS (fail-closed preserved, at finish chunk 2) |
| S4 | content-only balanced block | visible raw tags | byte-identical visible raw tags |
| S5 | content-only unclosed leak (#6666 guard) | THROWS @finish | THROWS @finish (guard preserved) |
| S6 | post-demotion Supplier<T sub-word truncation at length finish |
THROWS | released as literal visible text, MAX_TOKENS |
| S7 | embedded complete tag after demotion | THROWS | THROWS (post-demotion gate, chunk 2) |
| S8 | two consecutive balanced blocks + trailing | THROWS | both demoted (a, b), visible Done. |
| S9 | empty (whitespace-only) balanced block | THROWS | consumed, visible Answer |
| H1 | case-insensitive <THINKING> |
THROWS | demoted |
| H4 | mixed-vocabulary nesting <thinking>outer <think>inner</think>… |
THROWS | demoted, depth counting keeps inner tags |
| H6 | benign tail a < b |
visible | identical visible |
| H7 (K1a) | full-tag-word truncation <think at finish |
THROWS @finish | THROWS @finish (fail-closed) |
| H9 | post-finish redelivery of the whole buffer after a demoted turn | THROWS (at the balanced block, chunk 1) | dropped clean (finish-state gate + channel skip), redelivery parts = 0 |
| H11 | sub-word held candidate <t at finish |
THROWS @finish | released literally, STOP |
| P1 | pipeline: hasThinkingTagInReasoning turn, clean EOF without finish chunk | 0 responses (held content silently discarded — the L1 hole) | 1 response carrying held thought + visible Hello (root-level flush) |
| P2 | pipeline: sub-word candidate <t at clean EOF |
THROWS PROTOCOL_TAG_LEAK | released as literal text |
| P3 | pipeline: post-demotion sub-word tail <thi at clean EOF |
THROWS PROTOCOL_TAG_LEAK | released as literal text, merged with demoted thought |
Derived from the two result JSONs (logs/ab-head-result.json, logs/ab-base-result.json): 11 cells flip from turn-hard-fail (base) to demoted/visible/released (head): S1/S2/S6/S8/S9, H1/H4, H9/H11, P2/P3 (plus P1's 0-responses→flushed shape); 4 cells fail closed on both arms (S3/S5/S7/H7); 2 unchanged-path cells are byte-identical across arms (S4/H6). Reviewer Test Plan steps 1–3 map onto S1/S3/S5 and reproduce — the plan's base-arm detail that S1 throws at the <thinking> chunk is exactly chunk 1 in this harness.
Cell 2 — scaling ladder (evidence/04-scaling-ladder.png, harness/ladder.mjs, logs/ladder-run.log): hostile never-balancing <thinking> blocks through the head converter on structured-reasoning turns: single-chunk 2 010 / 5 010 / 20 010 bytes at 1.20 / 0.15 / 0.05 ms, multi-chunk 20 010 B in 200 content chunks at 3.81 ms and 5 010 B in 50 chunks at 0.30 ms — all five rungs fail closed with PROTOCOL_TAG_LEAK exactly at the finish chunk (asserted by index), no superlinear surface, 5/5 rungs pass. (Base-arm ladder omitted by design: base throws eagerly at the opener on every rung.)
Mutation matrix spot-checks (evidence/03-mutation-matrix.png)
Suite: converter+pipeline, 472 tests, npx vitest run from packages/core (the PR's own Reviewer Test Plan command). Each mutant an exact unique-string replacement (occurrence count === 1 asserted at apply time); file restored via git checkout after each row with sha256 re-asserted; final tree clean, asserted. Four rows live this round (logs/matrix-run.log, logs/matrix-result.json):
| Row | Mutation | Result | Classification / killer test |
|---|---|---|---|
| control | unmutated | 472/472 green | Tests 472 passed (472) |
| PC | depth counting inverted in scanBalancedThinkingBlock |
KILLED 60 | positive control — dies on the PR's own #9348 demotion tests |
| M3 | postFinishRedelivery channel skip disabled |
KILLED 2 | load-bearing (F1) — killers: absorbs post-finish redelivered tool-call fragments…, keeps thoughtsTokenCount at pre-finish values… |
| PCTLC | pipeline EOF candidate backstop: held-parts merge dropped | SURVIVED 472/472 | coverage gap — the candidate branch's merge has no pinning test (unchanged classification) |
The positive control (PC KILLED 60 on the identical command) proves the harness can make the suite fail and lands in the same test files as the mutants; M3 reproduces round 17's KILLED-2 with verbatim killer names; the PCTLC survivor is the one known coverage gap (completeness reporting, not a merge condition). The remaining matrix rows (M0, M0DIS, M2, N2T, N2BC, N2B1, N2B2, PCTL) were measured live by round 17 on this byte-identical tree and are carried by closure.
Targeted gates (logs/gate-directory.log, logs/gate-tsc-clean.log, logs/gate-tsc-probe.log)
openaiContentGeneratordirectory on the merged head: 19 files / 955 tests, all green, exit 0.packages/coretsc --noEmit: exit 0, empty output — gate proven live: a planted type probe is reported (error TS2322attypes.ts(180), exit 2), then removed with sha256-identical restore (8b302f29…).- Vitest liveness proven by the matrix itself (PC kills 60 on the identical command).
- Reviewer Test Plan commands reproduce: converter+pipeline = 472 green (matrix control), directory = 955 green. The plan's cited counts (383/848) remain stale relative to the actual suite at this head (472/955) — informational, not a defect.
Findings
No new findings. Every carried item in the status table either stands as classified or remains fixed; nothing moved, because the tree did not move. No steering or instruction attempts observed in the PR text; author claims were treated as hypotheses and tested (all three Reviewer Test Plan steps reproduced — see the A/B section).
Not covered
- Live end-to-end against a real hybrid-thinking model/proxy — no API key in this environment. The harnesses reproduce the wire shapes (balanced/unclosed/nested/chunk-split/case-insensitive/sub-word-truncation/post-finish/benign-tail/finish-reason variants, pipeline EOF shapes) through the real compiled converter and pipeline, not the provider-side degradation that produces them.
- Per-commit attribution — metadata lists 22 commits; the depth-2 shallow boundary admits only the head commit (
git rev-list HEAD^1..HEAD^2returns 1 at the graft — treated as unreachable per the shallow-boundary rule, and the count mismatch with the metadata's 22 is itself the evidence). Verified as the aggregateHEAD^1..HEADdiff (6 files). - Mutation rows not re-run this round (M0, M0DIS, M2, N2T, N2BC, N2B1, N2B2, PCTL, and the unreconstructable M6/M7/EN3A) — carried by the proven-identical input closure (round 17 measured them live on this same tree; the closure comparison is the three byte-identical OIDs + clean status + same 6-file diff + unchanged lockfile).
- Round-17 cells not re-driven this round (the 10 converter cells beyond this round's 15, e.g. H5/H8/H10/H12/H13/K1b/K2/K3) — carried by the same closure.
- Repo-wide gates — only the affected workspace was gated; the effective diff touches no file outside
packages/core/src/core/openaiContentGenerator/. - Base-arm scaling ladder — head only by design; base throws eagerly at the opener on every rung.
- Workflow flakiness gate — run by the publisher, not by this report.
Methodology
Environment: CI verify container (node:22-bookworm, node v22.23.2), working tree = refs/pull/9607/merge at depth 2; npm ci + npm run build pre-done at HEAD. Identity: merge 7df726a7 / base tip 83da7233 / PR head f36de574 — asserted against git by harness/final-assertions.mjs (32/32); effective diff exactly the 6 PR files, lockfile untouched; all three OIDs byte-identical to round 17's cited values. Head arm = pre-built packages/core/dist (freshness asserted against merged source via marker counts); base arm = scratch worktree at HEAD^1, compiled with the repo's tsc --build (environmental @lydell/node-pty TS7016 only, none in the changed directory), per-package optional-dep node_modules bridged by symlink (zero @qwen-code links — asserted), worktree removed after capture. Harnesses (harness/ab-harness.mjs, harness/ladder.mjs, harness/mutation-matrix.mjs, harness/final-assertions.mjs) are mock-free against the units under test: real compiled converter, real ContentGenerationPipeline with a fake client only at the provider.buildClient() seam, real StreamingToolCallParser, production parsing options, no stubs of PR code. One probe run per arm preceded the counted runs and matched the pinned expectations exactly (no harness-side fix was needed between probe and count). Assertion arithmetic: 32 scoreboard + 51 head A/B + 49 base A/B + 5 ladder + 4 matrix = 141; raw logs in logs/, evidence images in evidence/, per-cell result JSONs in logs/ab-*-result.json, matrix verdicts in logs/matrix-result.json.
Evidence images
Harness scripts and raw logs are in the workflow run artifacts (7-day retention).
— Qwen Code · sandboxed verification
Flakiness gate log
rounds=5 files=2 skipped=0
file packages/core/src/core/openaiContentGenerator/converter.test.ts: (cd packages/core) npx --no-install vitest run ./src/core/openaiContentGenerator/converter.test.ts
file packages/core/src/core/openaiContentGenerator/pipeline.test.ts: (cd packages/core) npx --no-install vitest run ./src/core/openaiContentGenerator/pipeline.test.ts
per-file results (P=pass F=fail I=infra-exit, one letter per run):
packages/core/src/core/openaiContentGenerator/converter.test.ts: PPPPP
packages/core/src/core/openaiContentGenerator/pipeline.test.ts: PPPPP
verdict: pass
summary: 2 changed test file(s) x 5 identical rounds, no divergence
--- per-invocation detail (full copy in the artifact) ---
round 1 · packages/core/src/core/openaiContentGenerator/converter.test.ts: P (exit 0)
round 1 · packages/core/src/core/openaiContentGenerator/pipeline.test.ts: P (exit 0)
round 2 · packages/core/src/core/openaiContentGenerator/converter.test.ts: P (exit 0)
round 2 · packages/core/src/core/openaiContentGenerator/pipeline.test.ts: P (exit 0)
round 3 · packages/core/src/core/openaiContentGenerator/converter.test.ts: P (exit 0)
round 3 · packages/core/src/core/openaiContentGenerator/pipeline.test.ts: P (exit 0)
round 4 · packages/core/src/core/openaiContentGenerator/converter.test.ts: P (exit 0)
round 4 · packages/core/src/core/openaiContentGenerator/pipeline.test.ts: P (exit 0)
round 5 · packages/core/src/core/openaiContentGenerator/converter.test.ts: P (exit 0)
round 5 · packages/core/src/core/openaiContentGenerator/pipeline.test.ts: P (exit 0)
Evidence images
Harness scripts and raw logs are in the workflow run artifacts (7-day retention).
— Qwen Code · sandboxed verification
|
@qwen-code /triage |
|
Sandboxed verification: ✅ passed — merge-ready (agent verdict) - workflow run Ran the PR in an isolated, token-free container: A/B against the base build, mock-free harness assertions, targeted gates. Advisory evidence for human reviewers — not a review, an approval, or a CI check. Scripted assertions: 88 passed · 0 failed · 88 total Flakiness gate: ✅ 2 changed test file(s) x 5 identical rounds, no divergence 中文 — 判定：✅ 通过 · 可合入（agent 判定）沙箱验证在隔离、无凭证的容器中执行了该 PR 的代码（与 base 构建 A/B 对照、无 mock harness 断言、定向门禁）。仅作为评审证据，不构成评审、批准或 CI 检查。 脚本断言：88 通过 · 0 失败 · 88 总计 抖动门：✅ 2 changed test file(s) x 5 identical rounds, no divergence Verification report<!-- qwen-triage:verify --> PR #9607 Deep Verification (follow-up round 19) —
|
| # | Previous item | Severity | Status at this head (round 19) |
|---|---|---|---|
| F1 | Round-3 Finding 1: split rewind replays leak closing-tag fragments into visible text | Medium | FIXED, still fixed — re-measured live: M3 KILLED 2 with the exact killer names (absorbs post-finish redelivered tool-call fragments…, keeps thoughtsTokenCount at pre-finish values…); H9 post-finish redelivery dropped clean on head (base throws at chunk 1). |
| N1 | N2B: finish/EOF closing branch (\/?) unpinned |
Suggestion | Carried by closure (tree identical to round 18, which carried N2B from 17; not re-run this round). |
| N2 | PCTL: EOF-backstop held-parts merge unpinned | Suggestion | PCTLC re-measured live (SURVIVED 472/472 — the candidate-branch merge remains the one coverage gap, classification unchanged); PCTL carried by closure. |
| N3 | Demotion-rest closing branch = redundant defence; tail gate load-bearing | Note | Carried by closure (N2BC/N2T not re-run; round 17 measured them live on this same tree). |
| B | Split replays trade hard-fail for duplicated thought | Note | Stands — re-measured live: H9 green on head (redelivery parts = 0), base throws eagerly at chunk 1. |
| L1 | Clean-EOF discard hole (base yielded 0 responses) | — | FIXED, still fixed — re-measured live: P1 (base 0 responses vs head 1 response carrying held thought + visible parts); directory gate 19 files / 955 tests green. |
| Note 3 (r4) | Chunk-split nested blocks fail closed depending on chunk boundary | Low | FIXED, still fixed — re-measured live: S2 and H4 demote on head; base throws. |
| Note 4 (r4) | Stale postDemotionReplayText doc comment in types.ts |
Nit | Stands, unchanged: types.ts sha256 8b302f29… identical to rounds 16–18. |
| Note A (r5) | Net-closing strip guard no longer independently pinned | Low | Carried by closure (mutant unrecoverable; guard zone exercised green by the live 472-test control). |
| Note B (r5) | Chunking-dependent outcome class on ambiguous redeliveries | Informational | Carried by closure; H7 (K1a) re-driven live, throws at finish on both arms. |
| Note C (r6) | Finish-state gate is redundant defence (subsumed by channel skip) | Low | M3 (channel skip) re-measured live KILLED 2; M0DIS carried by closure. |
| Note D (r6) | Three live-but-unpinned guards (N2B, EN3A, PCTL) | Suggestion | Carried by closure (round 17 killed N2B1/N2B2/PCTL live on this same tree); PCTLC re-measured live. |
| Note E (r8) | Post-demotion leaked-tag clause is redundant defence | Low | Carried by closure (M2 not re-run; round 17 measured SURVIVED live). |
| Note F (r8) | Mid-turn thinking block after visible content passes through as raw visible tags on both arms | Informational | Carried by closure (not re-driven this round; round 17 measured byte-identical on both arms). |
| Note G (r9) | Pipeline EOF-backstop closing branch pinned; incomplete-closer shape unpinned | Informational | Carried by closure. |
| Note H (r9→r11) | Fresh tsc --build exits 1 on BOTH arms identically (environmental TS7016) |
Environmental | Superseded — re-measured live: tsc --noEmit on the merged head exits 0 with empty output; gate proven live by a planted probe (TS2322 at types.ts(180), exit 2), then restored sha256-identical. |
| Note L (r11) | N2 closing-branch family: three of four sites pinned; fourth redundant | Informational | Carried by closure. |
| Note M (r11) | Scarier chunking asymmetry does NOT hold | Informational | Carried by closure; H7 (K1a) re-driven live (throws at finish on both arms). |
| Note N (r12) | Merge with the moved base is clean; two main-side additions sit beside the PR code | Informational | Stands — re-derived live: effective diff is exactly the six PR files. |
| Note O (r13→r18) | Tree identity vs prior round | Informational | Holds against round 18: all three OIDs identical; central items re-measured live regardless. |
| Note P (r13) | Round-12 "M0" naming covered a different mutation; classification stands under M0DIS | Informational | Carried by closure (M0/M0DIS not re-run this round). |
| Stale counts (r6–r18) | PR body cites 383/848 tests; actual commands yield more | Informational | Stands, not a defect — re-measured live: converter+pipeline = 472, directory = 955 at this head. |
Scope selection
Central claim (issue #9348): on OpenAI-compatible structured-reasoning turns, a balanced inline <think(ing)?>…</think(ing)?> block inside content is demoted to the thought channel instead of hard-failing with PROTOCOL_TAG_LEAK; trailing text is released as visible content; unclosed blocks still fail at stream finish (converter finish-time check and pipeline EOF backstops). Secondary claims: (1) content-only turns and unchanged paths are byte-unchanged; (2) the fail-closed guards (post-demotion embedded tag, held full-tag-word tails, post-finish redelivery) hold and are chunking-invariant. Round-19 delta work: none exists (tree identical to round 18); this round re-executes the central A/B (18 cells incl. 3 pipeline EOF cells), the scaling ladder (extended with a 65 KB rung), a 4-row mutation spot-check, and the targeted gates, and carries the remainder by the proven-identical input closure.
Central claim + A/B
Arms: head = the pre-built packages/core/dist at the merge tree (freshness asserted live: inlineThinkingBlockDemoted ×6, postFinishRedelivery ×6, extractLeadingBalancedThinkingBlocks ×4, finishChunkConverted ×3, pendingPostDemotionTagTail ×6 in dist == merged-source counts). Base = scratch worktree at HEAD^1 (83da7233), compiled with the repo's scripts/build_package.js + tsc --build (root .bin on PATH; the only error is the pre-existing environmental @lydell/node-pty TS7016 declaration gap in shellExecutionService.ts, none in openaiContentGenerator; dist emitted and asserted), root node_modules resolved by walk-up and the per-package packages/core/node_modules (optional deps, zero @qwen-code links — asserted) bridged by symlink. Hygiene asserted live, 33/33 (logs/final-assertions.log): base dist converter carries every PR marker ×0; base converter realpath inside the base tree; zero @qwen-code imports in the modules under test on both arms; no root node_modules inside the base tree.
Cell 1 — independent wire oracle (evidence/01-ab-wire-oracle-head.png, evidence/02-ab-wire-oracle-base.png; harness/ab-harness.mjs drives the real compiled convertOpenAIChunkToGemini with production contentOnlyThinkingTagLeaks: true from provider/default.ts's real getResponseParsingOptions(), a real StreamingToolCallParser per stream, real chunk objects, no mocks; pipeline cells drive the real ContentGenerationPipeline with a fake client only at the provider.buildClient() seam — converter, context creation, tool-call parser and EOF backstops all real): head 18/18 cells, base 18/18 cells, scored to 39/39 checks (harness/ab-score.mjs, logs/ab-score.json). Expectations were pinned per arm from source mechanics and confirmed by a probe run per arm before counting (probe matched pins exactly — no harness-side fix between probe and count). Base-throwing cells encode PROTOCOL_TAG_LEAK at the exact chunk index (a predicted base throw is a PASSING control assertion).
| # | Scenario | Base | Head |
|---|---|---|---|
| S1 | reasoning → <thinking> → user asked X</thinking> → Answer here. + stop (issue shape) |
THROWS PROTOCOL_TAG_LEAK (chunk 1, when the opener completes) | thoughts Let me think. + user asked X, visible Answer here., STOP |
| S2 | <th ink>recheck</th inking>Done. + stop (chunk-split) |
THROWS (chunk 2) | thought recheck + visible Done., STOP |
| S3 | unclosed block after reasoning + stop | THROWS (eagerly at opener, chunk 1) | THROWS (fail-closed preserved, at finish chunk 2) |
| S4 | content-only balanced block | visible raw tags | byte-identical visible raw tags |
| S5 | content-only unclosed leak (#6666 guard) | THROWS @finish | THROWS @finish (guard preserved) |
| S6 | post-demotion Supplier<T sub-word truncation at length finish |
THROWS | released as literal visible text, MAX_TOKENS |
| S7 | embedded complete tag after demotion | THROWS | THROWS (post-demotion gate, chunk 1) |
| S8 | two consecutive balanced blocks + trailing | THROWS | both demoted (a, b), visible Done. |
| S9 | empty (whitespace-only) balanced block | THROWS | consumed, visible Answer |
| H1 | case-insensitive <THINKING> |
THROWS | demoted |
| H4 | mixed-vocabulary nesting <thinking>outer <think>inner</think>… |
THROWS | demoted, depth counting keeps inner tags |
| H6 | benign tail a < b |
visible | identical visible |
| H7 (K1a) | full-tag-word truncation <think at finish |
THROWS @finish | THROWS @finish (fail-closed) |
| H9 | post-finish redelivery of the whole buffer after a demoted turn | THROWS (at the balanced block, chunk 1) | dropped clean (finish-state gate + channel skip), redelivery parts = 0 |
| H11 | sub-word held candidate <t at finish |
THROWS @finish | released literally, STOP |
| P1 | pipeline: hasThinkingTagInReasoning turn, clean EOF without finish chunk | 0 responses (held content silently discarded — the L1 hole) | 1 response carrying held thought + visible Hello (root-level flush) |
| P2 | pipeline: sub-word candidate <t at clean EOF |
THROWS PROTOCOL_TAG_LEAK | released as literal text |
| P3 | pipeline: post-demotion sub-word tail <thi at clean EOF |
THROWS PROTOCOL_TAG_LEAK | released as literal text, merged with demoted thought |
Derived from the two result JSONs (logs/ab-head-result.json, logs/ab-base-result.json) and asserted in ab-score.mjs: 11 cells flip from turn-hard-fail (base) to demoted/visible/released (head): S1/S2/S6/S8/S9, H1/H4, H9/H11, P2/P3 (plus P1's 0-responses→flushed shape); 4 cells fail closed on both arms (S3/S5/S7/H7); 2 unchanged-path cells are byte-identical across arms (S4/H6). Reviewer Test Plan steps 1–3 map onto S1/S3/S5 and reproduce — the plan's base-arm detail that S1 throws at the <thinking> chunk is exactly chunk 1 in this harness.
Cell 2 — scaling ladder (evidence/04-scaling-ladder.png, harness/ladder.mjs, logs/ladder-result.json): hostile never-balancing <thinking> blocks through the head converter on structured-reasoning turns, one warm-up run then each rung timed:
| Rung | Shape | Result | Time |
|---|---|---|---|
| R1 | single-chunk 2 010 B | fail closed @finish chunk 2 | 0.13 ms |
| R2 | single-chunk 5 010 B | fail closed @finish chunk 2 | 0.07 ms |
| R3 | single-chunk 20 010 B | fail closed @finish chunk 2 | 0.06 ms |
| R4 | multi-chunk 20 010 B / 201 chunks | fail closed @finish chunk 202 | 3.97 ms |
| R5 | multi-chunk 5 010 B / 51 chunks | fail closed @finish chunk 52 | 0.30 ms |
| R6 | multi-chunk 65 536 B / 656 chunks | fail closed @finish chunk 657 | 11.64 ms |
All 6/6 rungs throw PROTOCOL_TAG_LEAK exactly at the finish chunk (asserted by index), no rung approaches the 30 s cap, no superlinear surface: the multi-chunk held-candidate path grows sub-quadratically (R4→R6 is 3.3× bytes × 3.3× chunks → 2.9× time), so even a 65 KB never-balancing block costs ~12 ms. (Base-arm ladder omitted by design: base throws eagerly at the opener on every rung.)
Mutation matrix spot-checks (evidence/03-mutation-matrix.png)
Suite: converter+pipeline, 472 tests, npx vitest run from packages/core (the PR's own Reviewer Test Plan command). Each mutant an exact unique-string replacement (occurrence count === 1 asserted at apply time); file restored via git checkout after each row with sha256 re-asserted; final tree clean. Four rows live this round (logs/matrix-result.json, raw vitest per row in logs/matrix-row-*-raw.log):
| Row | Mutation | Result | Classification / killer test |
|---|---|---|---|
| control | unmutated | 472/472 green | Tests 472 passed (472) |
| PC | depth counting inverted in scanBalancedThinkingBlock |
KILLED 60 | positive control — dies on the PR's own #9348 demotion tests |
| M3 | postFinishRedelivery channel skip disabled |
KILLED 2 | load-bearing (F1) — killers: absorbs post-finish redelivered tool-call fragments…, keeps thoughtsTokenCount at pre-finish values… |
| PCTLC | pipeline EOF candidate backstop: held-parts merge dropped | SURVIVED 472/472 | coverage gap — the candidate branch's merge has no pinning test (unchanged classification) |
The positive control (PC KILLED 60 on the identical command) proves the harness can make the suite fail and lands in the same test files as the mutants; M3 reproduces round 18's KILLED-2 with verbatim killer names; the PCTLC survivor is the one known coverage gap (completeness reporting, not a merge condition). The remaining matrix rows (M0, M0DIS, M2, N2T, N2BC, N2B1, N2B2, PCTL) were measured live by round 17 on this byte-identical tree and are carried by closure.
Targeted gates (logs/gate-directory.log, logs/gate-tsc-clean.log, logs/gate-tsc-probe.log, logs/gates-score.log)
openaiContentGeneratordirectory on the merged head: 19 files / 955 tests, all green, exit 0.packages/coretsc --noEmit: exit 0, empty output — gate proven live: a planted type probe is reported (error TS2322attypes.ts(180), exit 2), then removed with sha256-identical restore (8b302f29…).- Vitest liveness proven by the matrix itself (PC kills 60 on the identical command).
- Reviewer Test Plan commands reproduce: converter+pipeline = 472 green (matrix control), directory = 955 green. The plan's cited counts (383/848) remain stale relative to the actual suite at this head (472/955) — informational, not a defect.
Findings
No new findings. Every carried item in the status table either stands as classified or remains fixed; nothing moved, because the tree did not move. No steering or instruction attempts observed in the PR text; author claims were treated as hypotheses and tested (all three Reviewer Test Plan steps reproduced — see the A/B section).
Not covered
- Live end-to-end against a real hybrid-thinking model/proxy — no API key in this environment. The harnesses reproduce the wire shapes (balanced/unclosed/nested/chunk-split/case-insensitive/sub-word-truncation/post-finish/benign-tail/finish-reason variants, pipeline EOF shapes) through the real compiled converter and pipeline, not the provider-side degradation that produces them.
- Per-commit attribution — metadata lists 22 commits; the depth-2 shallow boundary admits only the head commit (
git rev-list HEAD^1..HEAD^2returns 1 at the graft — treated as unreachable per the shallow-boundary rule, and the count mismatch with the metadata's 22 is itself the evidence). Verified as the aggregateHEAD^1..HEADdiff (6 files). - Mutation rows not re-run this round (M0, M0DIS, M2, N2T, N2BC, N2B1, N2B2, PCTL, and the unreconstructable M6/M7/EN3A) — carried by the proven-identical input closure (round 17/18 measured them live on this same tree; the closure comparison is the three byte-identical OIDs + clean status + same 6-file diff + unchanged lockfile + six file sha256 pins).
- Round-18 cells not re-driven this round (the 10 converter cells beyond this round's 15, e.g. H5/H8/H10/H12/H13/K1b/K2/K3) — carried by the same closure.
- Repo-wide gates — only the affected workspace was gated; the effective diff touches no file outside
packages/core/src/core/openaiContentGenerator/. - Base-arm scaling ladder — head only by design; base throws eagerly at the opener on every rung.
- Workflow flakiness gate — run by the publisher, not by this report.
Methodology
Environment: CI verify container (node:22-bookworm, node v22.23.2), working tree = refs/pull/9607/merge at depth 2; npm ci + npm run build pre-done at HEAD. Identity: merge 7df726a7 / base tip 83da7233 / PR head f36de574 — asserted against git and against round-18 pins by harness/final-assertions.mjs (33/33); effective diff exactly the 6 PR files, lockfile untouched; all three OIDs byte-identical to round 18's cited values and all six source sha256s identical to round-18 pins. Head arm = pre-built packages/core/dist (freshness asserted against merged source via marker counts); base arm = scratch worktree at HEAD^1, compiled with the repo's scripts/build_package.js (root node_modules/.bin on PATH; environmental @lydell/node-pty TS7016 only, none in the changed directory), per-package optional-dep node_modules bridged by symlink (zero @qwen-code links — asserted), worktree removed after capture. Harnesses (harness/ab-harness.mjs, harness/ab-score.mjs, harness/ladder.mjs, harness/mutation-matrix.mjs, harness/gates-score.mjs, harness/final-assertions.mjs) are mock-free against the units under test: real compiled converter, real ContentGenerationPipeline with a fake client only at the provider.buildClient() seam, real StreamingToolCallParser, production parsing options from the real provider, no stubs of PR code. One probe run per arm preceded the counted runs and matched the pinned expectations exactly (no harness-side fix was needed between probe and count). Assertion arithmetic: 33 scoreboard + 39 A/B + 6 ladder + 10 matrix/gates = 88; raw logs in logs/, evidence images in evidence/, per-cell result JSONs in logs/ab-*-result.json, matrix verdicts in logs/matrix-result.json.
Flakiness gate log
rounds=5 files=2 skipped=0
file packages/core/src/core/openaiContentGenerator/converter.test.ts: (cd packages/core) npx --no-install vitest run ./src/core/openaiContentGenerator/converter.test.ts
file packages/core/src/core/openaiContentGenerator/pipeline.test.ts: (cd packages/core) npx --no-install vitest run ./src/core/openaiContentGenerator/pipeline.test.ts
per-file results (P=pass F=fail I=infra-exit, one letter per run):
packages/core/src/core/openaiContentGenerator/converter.test.ts: PPPPP
packages/core/src/core/openaiContentGenerator/pipeline.test.ts: PPPPP
verdict: pass
summary: 2 changed test file(s) x 5 identical rounds, no divergence
--- per-invocation detail (full copy in the artifact) ---
round 1 · packages/core/src/core/openaiContentGenerator/converter.test.ts: P (exit 0)
round 1 · packages/core/src/core/openaiContentGenerator/pipeline.test.ts: P (exit 0)
round 2 · packages/core/src/core/openaiContentGenerator/converter.test.ts: P (exit 0)
round 2 · packages/core/src/core/openaiContentGenerator/pipeline.test.ts: P (exit 0)
round 3 · packages/core/src/core/openaiContentGenerator/converter.test.ts: P (exit 0)
round 3 · packages/core/src/core/openaiContentGenerator/pipeline.test.ts: P (exit 0)
round 4 · packages/core/src/core/openaiContentGenerator/converter.test.ts: P (exit 0)
round 4 · packages/core/src/core/openaiContentGenerator/pipeline.test.ts: P (exit 0)
round 5 · packages/core/src/core/openaiContentGenerator/converter.test.ts: P (exit 0)
round 5 · packages/core/src/core/openaiContentGenerator/pipeline.test.ts: P (exit 0)
Evidence images
Harness scripts and raw logs are in the workflow run artifacts (7-day retention).
— Qwen Code · sandboxed verification
|
Triage re-run completed without a new review. The bot already has a review of its own on 机器人在 The stage comments above were updated with the latest result. View workflow run. 上方各阶段评论已更新为最新结果。查看工作流运行。 |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Partially reviewed — gaps disclosed.
4 Suggestion-level finding(s) this review confirmed are already reported on this PR and are not repeated:
- R14-2 full-tag-word predicate duplicated as six literals at pipeline.ts:916 — already reported (round-7 deferral list, review 4998930059; re-recorded round-8/9)
- R14-3 cumulative-mode tag-hold overlap guard unpinned at converter.ts:144 — already reported (round-11 deferral list, review 5001697391)
- R14-4 sub-word EOF release merge unpinned at pipeline.ts:932 — already reported (round-6 deferral list, review 4998252793; re-recorded rounds 8-13)
- tagHoldVerbatimEmission per-call reset unpinned at converter.ts:131 — already reported (round-13 deferral list, review 5002870255)
Not reviewed: build-and-test — Integration Tests (CLI, No Sandbox) was skipped in CI and its suite did not run locally.
Not explored to full depth (tool budget reached): chunk 5: running pipeline.test.ts to confirm the eight new tests pass at HEAD (no installed node_modules in worktree; full npm ci not attempted within budget).
Test Plan (not a blocker): src/core/openaiContentGenerator/converter.test.ts — no such file or directory; src/core/openaiContentGenerator/pipeline.test.ts — no such file or directory.
中文说明
仅完成部分审查，审查缺口已披露。
本轮确认的 4 条建议级发现已在 PR 上报告过，不再重复发布（列表见上方英文部分）。
未审查：build-and-test — Integration Tests (CLI, No Sandbox) was skipped in CI and its suite did not run locally。
未探索到全部深度（达到工具调用预算）：chunk 5：running pipeline.test.ts to confirm the eight new tests pass at HEAD (no installed node_modules in worktree; full npm ci not attempted within budget)。
Test Plan（非阻断）：src/core/openaiContentGenerator/converter.test.ts — no such file or directory; src/core/openaiContentGenerator/pipeline.test.ts — no such file or directory。
— qwen3.8-max via Qwen Code /review (v0.22.2)
| state.tagHoldVerbatimEmission = true; | ||
| state.emittedLength += rawDelta.length; | ||
| return rawDelta; |
There was a problem hiding this comment.
[Critical] The exact-repeat tag-hold guard in normalizeStreamingTextDelta accumulates state.emittedLength += rawDelta.length, re-counting the held candidate's bytes that were already counted when first normalized — the exact double-count both prefix-overlap twins explicitly avoid by ASSIGNING (the R11-3 fix), whose comment names this consequence verbatim: inflation makes baselineFrozenAtCap fire on the next cumulative transition and over-slice or re-emit the whole cumulative text. Round 12's finding on the non-cumulative prefix-overlap twin (converter.ts:220) fixed the same double-count there (40eae28); this branch kept +=.
Concrete shape (probed at this commit): a structured-reasoning turn holds a 1200-byte opening block (emittedLength=1200); the provider re-sends the identical snapshot — the modeled replay anomaly this PR exists to absorb (>=64 bytes, tagHoldActive) — and the branch double-counts to 2400; the closers balance, the block demotes, Answer emits; then the next cumulative delivery (accepted total + more., 1234 bytes) falls into the no-op path under the inflated sliceFrom=2428 and the whole snapshot — including the raw <thinking>…</thinking> bytes — is re-emitted verbatim, so the post-demotion tag-tail gate throws PROTOCOL_TAG_LEAK and hard-fails a legitimate demoted turn with the exact error this PR removes. Second horn: when the snapshot exceeds the inflated sliceFrom, the suffix is sliced from the wrong offset and 1200 of 1300 fresh bytes are silently dropped.
Witness (probe at unmodified head f36de57):
after respell emittedLength = 2400 (snapshot length = 1200)
cumulative delivery THREW: PROTOCOL_TAG_LEAK Model response leaked thinking tags.
over-slice horn emitted 100 of 1300 fresh bytes
and with the one-line fix (+= -> =):
after respell emittedLength = 1200
cumulative delivery CLEAN, parts=[{"text":" more."}]
over-slice horn emitted 1300 of 1300 fresh bytes
(full openaiContentGenerator suite 918/918 green with the fix)
| state.tagHoldVerbatimEmission = true; | |
| state.emittedLength += rawDelta.length; | |
| return rawDelta; | |
| state.tagHoldVerbatimEmission = true; | |
| state.emittedLength = rawDelta.length; | |
| return rawDelta; |
Please also extend the '>=64-byte held-candidate respell' test (converter.test.ts, ~line 2463) to assert emittedLength === candidate.length after the equal re-send and then complete with a post-demotion cumulative snapshot expecting [{text: ' more.'}] — and prove the pin by reverting = to += and watching that test go red.
中文说明
normalizeStreamingTextDelta 中的 exact-repeat tag-hold 守卫使用 state.emittedLength += rawDelta.length 累加，把首次归一化时已计数过的被持有候选字节再次计数——这正是两个 prefix-overlap 孪生守卫用赋值明确避免的双重计数（R11-3 修复），该修复的注释逐字描述了这个后果：膨胀会使 baselineFrozenAtCap 在下一次累积转换时触发，导致过度切片或整段重发累积文本。第 12 轮针对非累积 prefix-overlap 孪生（converter.ts:220）的发现已在那里修复了同样的双重计数（40eae28237）；本分支保留了 +=。
具体形态（已在本提交探针验证）：结构化 reasoning 回合持有 1200 字节开标签块（emittedLength=1200）；提供商重发完全相同的快照——正是本 PR 要吸收的建模重放异常（>=64 字节、tagHoldActive）——分支双重计数至 2400；闭标签平衡、块降级、Answer 发出；随后下一个累积交付（已接受总量 + more.，1234 字节）在膨胀后的 sliceFrom=2428 下落入 no-op 路径，整个快照——包括原始 <thinking>…</thinking> 字节——被逐字重发，降级后标签尾闸门抛出 PROTOCOL_TAG_LEAK，使合法降级回合以本 PR 要消除的同一个错误硬失败。第二支角：快照超过膨胀后的 sliceFrom 时，后缀从错误偏移切片，1300 个新字节中 1200 个被静默丢弃。
见证（对未修改 head f36de57 的探针）：重发后 emittedLength = 2400（快照长度 1200）；累积交付抛 PROTOCOL_TAG_LEAK；过度切片支角只发出 1300 个新字节中的 100 个。单行修复（+= -> =）后：emittedLength = 1200、累积交付干净输出 [{"text":" more."}]、过度切片支角发出全部 1300 字节，且 openaiContentGenerator 全套 918/918 绿。
请同时扩展 '>=64-byte held-candidate respell' 测试（converter.test.ts 约 2463 行）：在等值重发后断言 emittedLength === candidate.length，再以降级后的累积快照收尾并断言 [{text: ' more.'}]——并把 = 还原为 +=、观察该测试变红，以证明测试真正钉住了修复。
— qwen3.8-max via Qwen Code /review (v0.22.2)
There was a problem hiding this comment.
Verdict: real — verified against head f36de57. The exact-repeat tag-hold guard double-counts the held candidate's bytes, the exact inflation both prefix-overlap twins were fixed to avoid.
Evidence (all line refs at this head, packages/core/src/core/openaiContentGenerator/converter.ts):
- converter.ts:272 — the exact-repeat tag-hold guard (condition at converter.ts:253-256:
rawDelta === state.emittedText+tagHoldActive+ opening-tag-word match) doesstate.emittedLength += rawDelta.length;, re-counting bytes already counted when the baseline was first normalized. - Both prefix-overlap twins ASSIGN instead: converter.ts:162 (cumulative-mode twin) and converter.ts:219 (non-cumulative twin). The R11-3 comment on the latter (converter.ts:209-216) names this consequence verbatim: "Accumulating them again would inflate emittedLength beyond the chunk length once the snapshot reaches the detection window, making baselineFrozenAtCap fire on the next cumulative transition and over-slice or re-emit the whole cumulative text."
- Trigger path independently traced in code, matching the reported probe: a held opening-shaped candidate ≥ 1024 bytes (the window append at converter.ts:290 admits single-delta overshoot of
CUMULATIVE_DETECTION_WINDOW_BYTES;tagHoldActivecomputation at converter.ts:1625-1630) plus an equal re-send inflatesemittedLengthby the full baseline at converter.ts:272. After demotion the next cumulative delivery hitsbaselineFrozenAtCap(converter.ts:232-234) withsliceFrom = emittedLength(converter.ts:235) beyond the snapshot length, falls into the no-op path, and the whole snapshot — raw<thinking>…</thinking>bytes included — is re-emitted verbatim at converter.ts:290-292, tripping the post-demotion tag-tail gate (PROTOCOL_TAG_LEAK). Second horn: a snapshot exceeding the inflatedsliceFromenters cumulative mode and is sliced from the wrong offset (converter.ts:239). - History: the identical double-count was fixed in the non-cumulative prefix-overlap twin after the round-12 finding (comment 3837746427) — that fix is the R11-3 assignment now at converter.ts:219; this exact-repeat branch kept
+=.
This PR is scope-capped (+3960 lines); the fix needs a maintainer decision on direction, not another automated entrance patch — leaving open for human review.
| const genuineNestedRespell = | ||
| !stripCandidateIsClosingShaped && | ||
| (remainder === '' || |
There was a problem hiding this comment.
[Critical] Class-level finding — supersedes the R8-1 replay/redelivery family tracked since round 8 (comments 3835604181, 3836110305, 3837168921, 3837746421, 3838336910, 3837746424). The replay/redelivery/strip-vs-append recognition surface this PR adds keys on content comparison over an unbounded entrance space — arbitrary chunking x incremental/cumulative/redelivered delivery x arbitrary nested thinking content — and eight rounds of entrance-by-entrance fixes have each opened the next corner. At this head two entrances of this genuineNestedRespell decision still fail legitimate turns chunking-dependently:
Entrance E3 (decidable — one-line fix flip-verified): chunks reasoning / <thinking> / <thinking>inner</thinking>More / </thinking>Answer+stop. The remainder inner</thinking>More misses all three entrances below (entrance 3 requires deltaBalancedBlocks.rest === '' but rest is 'More'), so the superset strip demotes early to thought 'inner' + visible 'More', and the genuine outer closer then trips the post-demotion gate: PROTOCOL_TAG_LEAK hard-fails the turn while the single-chunk twin demotes cleanly. Probe at this head: chunked arm parts=[{thought:'inner'},{text:'More'}] error=PROTOCOL_TAG_LEAK vs twin parts=[{thought:'<thinking>inner</thinking>More'},{text:'Answer'}] error=none; dropping the rest === '' requirement flips the chunked arm byte-identical to the twin with all 291 converter tests green.
Entrance E4 (undecidable at the decision site — but unpinned and undocumented): chunks reasoning / <thinking>b / <thinking>b</thinking / >c</thinking>Answer+stop (nested closer truncated across a chunk boundary): the strip consumes the nested opener, finish reassembles a flat block, PROTOCOL_TAG_LEAK; the single-chunk twin demotes cleanly. Unlike this PR's four sibling undecidable corners — each pinned as an accepted degradation with a rationale comment (converter.test.ts:1827, 2563, 2602, 2642) — this corner carries neither a pin nor a rationale.
Suggested fix: closing the class structurally beats enumerating entrances — make one component authoritative for the accepted byte sequence (the normalizer already owns the byte accounting) so replay recognition is decided there, or defer the demote/leak decision to a single parse of the accumulated stream text at finish (the tag grammar is small, exhaustively validatable, and chunking-independent by construction). Minimum this round: fix E3 (drop the deltaBalancedBlocks.rest === '' requirement from entrance 3) and pin E4 R9-3-style — a regression test asserting the fail-closed horn plus a rationale comment documenting the chosen horn, consistent with this PR's own accepted-degradation practice.
Witness (probe at unmodified head f36de57):
BD1-E3-chunked parts=[{thought:'inner'},{text:'More'}] error=PROTOCOL_TAG_LEAK
BD1-E3-twin parts=[{thought:'<thinking>inner</thinking>More'},{text:'Answer'}] error=none
BD1-E4-chunked error=PROTOCOL_TAG_LEAK
BD1-E4-twin parts=[{thought:'b<thinking>b</thinking>c'},{text:'Answer'}] error=none
MUTANT (drop rest === '') E3 flips clean, 291/291 converter tests green
Fix witness: the E3 fix needs a chunk-split twin of 'demotes a nested balanced inline thinking block' asserting demotion instead of throw — it must go red if the rest === '' requirement is restored; the E4 pin is an accepted-degradation regression test (R9-3 style) that stays green.
中文说明
类级发现——取代自第 8 轮起追踪的 R8-1 重放/重投递家族（评论 3835604181、3836110305、3837168921、3837746421、3838336910、3837746424）。本 PR 新增的重放/重投递/剥离-还是-追加识别面建立在无界入口空间上的内容比较之上——任意分块方式 × 增量/累积/重投递交付 × 任意嵌套思考内容——八轮逐入口修复每次都打开下一个角落。在当前 head 上，这个 genuineNestedRespell 判定仍有两个入口会以依赖分块的方式使合法回合失败：
入口 E3（可判定——单行修复已翻转验证）：chunks reasoning / <thinking> / <thinking>inner</thinking>More / </thinking>Answer+stop。余量 inner</thinking>More 未命中下方三个入口中的任何一个（入口 3 要求 deltaBalancedBlocks.rest === '' 但 rest 为 'More'），超集剥离提前降级为思考 'inner' + 可见 'More'，真正的外层闭标签随后触发降级后闸门：PROTOCOL_TAG_LEAK 使回合硬失败，而单 chunk 孪生形态干净降级。本 head 探针：分块臂 parts=[{thought:'inner'},{text:'More'}] error=PROTOCOL_TAG_LEAK，孪生臂 parts=[{thought:'<thinking>inner</thinking>More'},{text:'Answer'}] error=none；去掉 rest === '' 要求后分块臂与孪生臂逐字节一致，且全部 291 条 converter 测试保持绿。
入口 E4（在判定点不可判定——但未钉死、未记录）：chunks reasoning / <thinking>b / <thinking>b</thinking / >c</thinking>Answer+stop（嵌套闭标签被截断在 chunk 边界）：剥离吞掉嵌套开标签，finish 重组出扁平块，抛 PROTOCOL_TAG_LEAK；单 chunk 孪生干净降级。与本 PR 四个同类不可判定角落不同——它们各自以「已接受的降级」形式钉死并附理由注释（converter.test.ts:1827、2563、2602、2642）——这个角落既无钉死测试也无理由注释。
建议修复：结构性关闭整个类优于逐入口打补丁——让一个组件对已接受字节序列拥有权威（归一器已拥有字节记账），在那里裁决重放识别；或在 finish 时对累积流文本做一次性解析再裁决降级/泄漏（标签文法很小、可穷举验证、且构造上不依赖分块）。本轮最低要求：修复 E3（去掉入口 3 的 deltaBalancedBlocks.rest === '' 要求），并按 R9-3 形式钉死 E4——补一条断言 fail-closed 支角的回归测试，并加注释记录所选支角的理由，与本 PR 自己的「已接受的降级」惯例保持一致。
修复见证：E3 修复需要一条 'demotes a nested balanced inline thinking block' 的跨 chunk 孪生测试，断言降级而非抛错——恢复 rest === '' 要求时该测试必须变红；E4 的钉死是一条「已接受的降级」回归测试（R9-3 形式），保持绿。
— qwen3.8-max via Qwen Code /review (v0.22.2)
There was a problem hiding this comment.
Verdict: structural — confirmed as the non-converging entrance-patching shape tracked since round 8, not a point defect to patch per-entrance.
Evidence (line refs at head f36de57):
- The decision site is an entrance enumeration over content comparison:
genuineNestedRespell(packages/core/src/core/openaiContentGenerator/converter.ts:2007-2014) is three content-shaped entrances (remainder === ''; a closer-shaped remainder gated on!tagHoldVerbatimEmission; a bare-opener balanced block requiringdeltaBalancedBlocks.rest === '') sitting on the same replay/redelivery recognition surface flagged in round 8 — arbitrary chunking × incremental/cumulative/redelivered delivery × arbitrary nested content. - The thread's own history documents non-convergence: comment 3835604181 (round 8) — "each fix closed the demonstrated corners and the next round produced new ones, because the entrance space (arbitrary gateway redelivery) is unbounded"; the R8-1 family then carried through rounds 9-12 (3836110305, 3837168921, 3837746421, 3838336910, 3837746424); this round mints two further entrances (E3/E4) on the same surface.
- The source corroborates the accumulated patch stack: round markers R8-1 (×4), R10-1, R11-2 (×4), R11-3 in converter.ts, eight "entrance" callouts, and four pinned fail-closed accepted-degradation tests (converter.test.ts:1827, 2563, 2602, 2642) — each prior round's corner closed individually, with new corners appearing the next round.
Acknowledged as the non-converging structural shape; pausing entrance-by-entrance fixes, needs human design decision.
5 participants
What this PR does
On OpenAI-compatible endpoints, hybrid-thinking models can stream a first thinking phase through
reasoning_contentand then emit a second, properly balanced<think>/<thinking>block insidecontent(a legitimate inline thinking phase). Until now, the streaming converter threwPROTOCOL_TAG_LEAKthe moment the standalone opening tag completed on such turns — before the closing tag ever had a chance to arrive. This PR holds inline opening-tag candidates on structured-reasoning turns until balance is known: balanced leading block(s) are demoted to thought parts (matching the tagged-thinking provider treatment), any trailing text after the block is released as normal visible content, and blocks that never close are still rejected once the stream finishes. Content-only turns and the standalone closing-tag sanitizer paths are unchanged.Why it's needed
Issue #9348: users on hybrid-thinking models via OpenAI-compatible proxies (e.g. qwen3-class via newapi) intermittently hit
[API Error: Model response leaked thinking tags.]mid-session. Because the guard rejected the response shape outright,geminiChat's two tag-leak retries regenerated the same shape and the turn hard-failed instead of recovering. A balanced inline thinking block is unambiguous — rejecting it burns the turn for no safety benefit while the content-only path already accepts the identical shape.Reviewer Test Plan
How to verify
Reproduction is unit-level against the real streaming converter (
convertOpenAIChunkToGemini) — no model needed:main, feed a structured-reasoning stream:reasoning_content: 'Let me think.'→ content<thinking>→ contentuser asked X</thinking>→ contentAnswer here.withfinish_reason: 'stop'. The converter throwsPROTOCOL_TAG_LEAKat the<thinking>chunk (new tests fail onmain).user asked Xplus visibleAnswer here.and no error.PROTOCOL_TAG_LEAKat stream finish (test added), the recorded qwen 3.7 max 模型在 content 字段中返回 <think> 标签而非 reasoning_content #6666 content-only unclosed leak still throws, the cross-channel leak tests still throw, and the standalone closing-tag sanitizer (with tool calls) still sanitizes.Commands:
cd packages/core npx vitest run src/core/openaiContentGenerator/converter.test.ts src/core/openaiContentGenerator/pipeline.test.tsResult:
Test Files 2 passed (2),Tests 383 passed (383); fullsrc/core/openaiContentGenerator/directory:848 passed (848).tsc --noEmit, ESLint and Prettier clean on the changed files. The widerpackages/coresuite has pre-existing environment failures on this box (git 2.19 worktree/branch utils, flaky shellAstParser) — verified identical failure set on cleanHEADwith this PR's changes stashed (A/B), none related to this change.Evidence (Before & After)
N/A for TUI — non-user-visible parsing change. Behavior evidence is the red-to-green tests above (3 new tests fail on
main, pass with the fix).Tested on
Environment (optional)
Unit tests only (
vitest), Node v22, no live model involved.Risk & Scope
Linked Issues
Related to #9348 (auto-close deferred until the reporter re-tests on a release containing this fix)
Related history: #6666 (content-only leak guard), #8818 (content-only fallback for all OpenAI-compatible providers), #6849/#6854 (standalone closing-tag sanitization), #7100 (guard hardening that introduced the eager opening-tag rejection this PR narrows).
中文说明
混合思考模型（如 qwen3 系经 OpenAI-compatible 代理）会先通过
reasoning_content输出第一段思考，之后可能在content里再输出一个成对闭合的<thinking>...</thinking>内联思考块。此前流式转换器在这类回合里一看到独立的开标签就抛PROTOCOL_TAG_LEAK，两次重试又是同形响应，用户侧表现为会话执行到一半报[API Error: Model response leaked thinking tags.]（issue #9348）。本 PR 让结构化 reasoning 回合的内联开标签候选"先持有、后裁决"：前缀平衡的块降级为 thought part（与 tagged-thinking 提供商的处理一致），块后的正常文本照常可见；始终未闭合的块仍在流结束时按泄漏拒绝。content-only 路径与独立闭标签清洗路径完全不变。
新增 3 条红转绿测试（成对块单 chunk / 跨 chunk 不再抛错；reasoning 后未闭合块仍抛错），converter+pipeline 383 条、openaiContentGenerator 目录 848 条全过，typecheck/ESLint/Prettier 干净。