fix(core): tolerate repeated invalid model streams

[yiliang114]

What this PR does

Increase the independent retry budget for transient invalid model streams from two retries to four while preserving the existing linear backoff and cancellation behavior. Add regression coverage for recovery on the fifth attempt after four invalid streams, as well as correct telemetry when the retry budget is exhausted.

Why it's needed

DashScope and some OpenAI-compatible providers occasionally return semantically incomplete streaming responses, including empty content, missing finish reasons, or empty choices containing only usage metadata. The client already detects these anomalies correctly, but the existing two-retry budget provides only a six-second recovery window and still surfaces InvalidStreamError after three consecutive invalid responses. This change expands the client-side recovery window without changing stream validation rules or claiming to fix the underlying server-side behavior.

Reviewer Test Plan

How to verify

Run the core streaming tests and confirm that four consecutive empty responses with a STOP finish reason are retried and followed by a successful fifth response. Verify that conversation history contains only the user message and final model response. Also confirm that persistent invalid streams stop after five total attempts, with four retry telemetry events and one terminal failure event.

Evidence (Before & After)

Before: The new regression test failed after the third API attempt with Model stream ended with empty response text.

After: The fifth API attempt succeeds. The four retry telemetry events report backoff delays of 2s, 4s, 6s, and 8s. The complete affected test file passes with 217/217 tests.

Tested on

OS	Status
🍏 macOS	✅
🪟 Windows	⚠️
🐧 Linux	⚠️

Environment (optional)

Node.js v22.22.0. Invalid SSE responses were simulated at the protocol boundary using fresh async stream generators.

Risk & Scope

• Main risk or tradeoff: Persistent invalid responses may now take up to 20 seconds of additional backoff before the error is surfaced, compared with six seconds previously. The wait remains interruptible through AbortSignal.
• Not validated / out of scope: A deterministic live reproduction against DashScope was not possible because the upstream behavior is intermittent. This change does not investigate or modify server-side behavior.
• Breaking changes / migration notes: None.

Linked Issues

Resolves #6670

Related to #3888

中文说明

本 PR 的改动

将瞬态无效模型流的独立重试预算从 2 次提高到 4 次，同时保留现有线性退避和取消行为。补充回归测试，覆盖连续四次无效响应后第五次成功，以及持续失败时正确耗尽预算和上报 telemetry。

修改原因

DashScope 和部分 OpenAI-compatible provider 偶尔会返回语义不完整的流式响应，包括空内容、缺少 finish reason，或只有 usage metadata 的 empty choices。现有客户端可以正确识别这些异常，但两次重试只能提供约 6 秒恢复窗口，连续发生三次异常时仍会直接向用户暴露 InvalidStreamError。本变更扩大客户端恢复窗口，用于缓解这类偶发服务端响应问题；它不改变验证规则，也不宣称修复服务端根因。

Reviewer 测试计划

验证方法

运行 core streaming tests，确认连续四次返回空内容且带 STOP 的响应后，第五次有效响应能够正常返回，并且历史记录中只保留一次用户消息和最终模型响应。确认持续返回无效流时，总共尝试五次后停止，并记录四次 retry telemetry 和一次 terminal failure telemetry。

修改前后证据

修改前：新增回归测试在第三次 API 调用后抛出 Model stream ended with empty response text.

修改后：第五次 API 调用成功返回，四次 retry telemetry 分别记录 2s、4s、6s、8s 的退避时间；完整受影响测试文件 217/217 通过。

测试环境

OS	状态
🍏 macOS	✅
🪟 Windows	⚠️
🐧 Linux	⚠️

Node.js v22.22.0；在协议边界使用 fresh async stream generators 模拟无效 SSE 响应。

风险与范围

• 主要风险或权衡：持续无效响应时，报错前的最大额外等待从 6 秒增加到 20 秒；等待仍可通过 AbortSignal 中断。
• 未验证或超出范围：未对真实 DashScope 间歇性故障执行确定性 E2E；未调查或修改服务端行为。
• Breaking changes 或迁移说明：无。

关联 Issue

Resolves #6670

Related to #3888

[qwen-code-ci-bot]

Re-triage requested by @wenshao after author addressed review feedback.

Template ✓ — all required headings present, bilingual.

Problem: Observed, well-documented. DashScope intermittent empty-stream failures are tracked across #6670, #3888, #1700, and #6007. The regression test (should recover after four consecutive invalid streams) fails deterministically on the pre-PR code — throws InvalidStreamError on the 3rd attempt with the old 2-retry budget. This isn't theoretical hardening; the 2-retry window is genuinely too narrow for peak-hour DashScope instability.

Direction: Aligned. Client-side retry resilience for transient provider-side stream anomalies is core infrastructure — reliable model streaming is the product. The #6670 triage itself suggested increasing the retry budget.

Size: 127 production lines (79+48 in geminiChat.ts), 378 test lines (373+5 in geminiChat.test.ts). Well under any escalation threshold. Not applicable for maintainer awareness flag.

Approach: Grew appropriately beyond the initial one-line change. The original PR was maxRetries: 2 → 4. After review iterations, it now correctly separates retry budgets by error type — transientMaxRetries: 4 for empty/truncated streams, protocolTagLeakMaxRetries: 2 for model-output validation failures. Both the main send loop and the output-continuation path use independent counters. Stream validation was also moved before history push so failed attempts don't leave residue in conversation state. The scope increase is justified: a shared counter would let transient retries preemptively exhaust the protocol-leak budget (or vice versa), which would have been a subtle correctness bug. The docs/plans/ file from the initial submission has been dropped.

Moving on to code review. 🔍

中文说明

应 @wenshao 要求，在作者处理 review 反馈后重新分类。

模板 ✓ — 所有必需标题齐全，双语。

问题：已观测、有充分记录。 DashScope 间歇性空流故障在 #6670、#3888、#1700、#6007 中均有追踪。回归测试（should recover after four consecutive invalid streams）在改动前代码上可确定性失败——旧的 2 次重试预算在第 3 次尝试时抛出 InvalidStreamError。这不是理论性加固；2 次重试窗口对 DashScope 高峰期确实不够用。

方向：对齐。 瞬态 provider 端流异常的客户端重试弹性属于核心基础设施——可靠的模型流式传输就是产品本身。#6670 的分类本身就建议增加重试预算。

规模：127 行生产代码（geminiChat.ts 中 79+48），378 行测试代码（geminiChat.test.ts 中 373+5）。远低于任何阈值，无需维护者关注标记。

方案：从初始一行改动合理扩展。 原始 PR 是 maxRetries: 2 → 4。经过 review 迭代，现在按错误类型正确分离重试预算——空/截断流 transientMaxRetries: 4，模型输出校验失败 protocolTagLeakMaxRetries: 2。主发送循环和输出续写路径都使用独立计数器。流校验也被移到历史推入之前，失败的尝试不会在对话状态中留下残留。范围扩大是合理的：共享计数器会让瞬态重试提前耗尽 protocol-leak 预算（或反之），这将是一个隐蔽的正确性 bug。初始提交中的 docs/plans/ 文件已被移除。

进入代码审查 🔍

— Qwen Code · qwen3.7-max

[qwen-code-ci-bot]

Code Review

Independent proposal: Given DashScope intermittent empty streams exhausting a 2-retry budget, I'd increase maxRetries to 4, update existing test assertions, and add a regression test for recovery after multiple invalid streams. I'd also consider whether protocol-tag-leak retries should share the same budget — they're a different failure mode (model output validation vs. provider-side empty streams) and probably shouldn't.

Diff review (re-run against current head dce63cc): The PR matches and exceeds the independent proposal. It separates retry budgets by error type (transientMaxRetries: 4 / protocolTagLeakMaxRetries: 2) with independent counters in both the main send loop and the output-continuation path. The separation is correct — transient empty streams and protocol-tag leaks have different root causes and should not compete for the same budget.

Key correctness points verified:

• Counter independence. transientInvalidStreamRetryCount and protocolTagLeakRetryCount are declared at the top of the send loop and incremented only for their respective error types. The totalInvalidStreamRetryCount() closure correctly sums both for the RETRY event gate. Same pattern in the continuation path with transientRetryCount / protocolTagLeakRetryCount.
• Backoff stays linear per-counter. initialDelayMs * nextInvalidStreamRetryCount where nextInvalidStreamRetryCount is the per-type counter after increment. So a sequence of 2 transient + 1 protocol-leak gives delays 2s, 4s (transient), 2s (protocol-leak) — each type's backoff is independent. This is correct.
• Stream validation moved before history push. The InvalidStreamError throw now fires before this.history.push and before recordAssistantTurn. A failed attempt leaves no residue in conversation history or chat-recording JSONL. This fixes a subtle bug where --resume could re-inject a stale partial turn.
• Telemetry correctness. ContentRetryFailureEvent.total_attempts uses totalInvalidStreamRetryCount() + 1 (sum of both counters + initial attempt). ContentRetryEvent.attempt_number is per-type (0-indexed within that type's budget). Both are internally consistent.
• popPendingPartialAssistantTurn() called on every invalid-stream retry, cleaning up before the next attempt.

No correctness bugs, no stale references, no AGENTS.md violations. The code is straightforward — no over-abstraction, no drive-by refactors.

Reuse check: No new utility functions or abstractions. The retry pattern extends the existing structure in place.

Testing

Unit tests (worktree, PR head dce63cc):

$ cd packages/core && npx vitest run src/core/geminiChat.test.ts

 ✓ src/core/geminiChat.test.ts (225 tests) 11307ms

Test Files  1 passed (1)
     Tests  225 passed (225)
  Duration  16.78s

All 225 tests pass, including the 8 new/modified tests covering:

• Recovery on 5th attempt after 4 consecutive invalid streams (with backoff delay assertions: 2s/4s/6s/8s)
• Budget exhaustion after 5 total attempts (4 retry events + 1 terminal failure, final_error_type: NO_FINISH_REASON)
• Protocol-tag-leak budget stays at 2 (stops after 3 attempts), independent of transient budget
• Mixed error-type sequence (2 transient + 1 protocol-leak + recovery)
• AbortSignal during retry delay surfaces immediately
• Output-continuation: protocol-tag-leak budget unchanged at 2
• Output-continuation: mixed error types with independent budgets
• Usage-only empty stream retry doesn't record failed attempts to history

Real-scenario tmux test: not applicable. This addresses intermittent DashScope server-side behavior that cannot be triggered via CLI on demand. The maintainer @wenshao independently verified this PR locally (225/225 pass, typecheck clean, lint clean) and confirmed the new tests genuinely fail on pre-PR code.

中文说明

代码审查

独立方案： 针对 DashScope 间歇性空流耗尽 2 次重试预算的问题，我会将 maxRetries 增加到 4，更新测试断言，并补充多次无效流后恢复的回归测试。同时考虑 protocol-tag-leak 重试是否应共用同一预算——它们是不同的故障模式（模型输出校验 vs provider 端空流），可能不应该共享。

Diff 审查（对当前 head dce63cc 重新审查）： PR 匹配并超越了独立方案。按错误类型分离重试预算（transientMaxRetries: 4 / protocolTagLeakMaxRetries: 2），主发送循环和输出续写路径都使用独立计数器。分离是正确的——瞬态空流和 protocol-tag leak 有不同根因，不应竞争同一预算。

验证的关键正确性点：计数器独立（transient 和 protocol-leak 分别声明、分别递增）；退避按计数器线性计算；流校验移到历史推入之前；telemetry 字段内部一致；每次无效流重试前调用 popPendingPartialAssistantTurn()。

未发现正确性 bug、过时引用或 AGENTS.md 违规。

测试

单元测试（worktree，PR head dce63cc）： 225/225 通过，包括 8 个新增/修改的测试。

真实场景 tmux 测试：不适用。 此 PR 解决的是间歇性 DashScope 服务端行为，无法通过 CLI 按需触发。维护者 @wenshao 已独立在本地验证（225/225 通过，typecheck 干净，lint 干净），并确认新测试在改动前代码上确实失败。

— Qwen Code · qwen3.7-max

[qwen-code-ci-bot]

This PR has matured well across four review iterations. What started as a one-line constant bump evolved into a proper fix that separates retry budgets by error type — the kind of improvement you only see after someone actually reviews the code carefully. The author was responsive to every round of feedback without scope-creeping beyond the retry-budget problem.

The core change is sound: DashScope empty streams and protocol-tag leaks are genuinely different failure modes with different recovery profiles. Giving transient errors 4 retries (20s window) while keeping protocol-tag leaks at 2 (6s window) reflects that distinction. The stream-validation-before-history-push fix is a real correctness improvement — failed attempts no longer leave residue that could corrupt --resume.

Tests are thorough and genuinely pin the behavior: 8 new/modified tests, confirmed failing on pre-PR code (by both the CI triage and @wenshao's independent local verification), all passing on the PR head. The mixed-error-type and abort-during-delay tests cover edge cases that would otherwise be silent regressions.

My independent proposal was "increase the constant, maybe separate the budgets." The PR did exactly that plus the history-push ordering fix, which I wouldn't have caught. The implementation is straightforward — no over-abstraction, no speculative features.

One minor note: the PR body still says "217 tests" but the current head has 225. Not a concern, just a doc lag.

Approving. ✅

中文说明

这个 PR 在四轮 review 迭代中成熟得很好。从最初的一行常量修改进化成了按错误类型分离重试预算的正确修复——这种改进只有在认真审查代码后才会发现。作者对每轮反馈都积极响应，且没有超出重试预算问题的范围。

核心改动合理：DashScope 空流和 protocol-tag leak 是真正不同的故障模式，有不同的恢复特征。瞬态错误给 4 次重试（20 秒窗口），protocol-tag leak 保持 2 次（6 秒窗口），反映了这种区别。流校验前移到历史推入之前是一个真正的正确性改进——失败的尝试不再留下可能污染 --resume 的残留。

测试全面且真正锁定行为：8 个新增/修改测试，在改动前代码上确认失败（CI 分类和 @wenshao 独立本地验证均确认），在 PR head 上全部通过。混合错误类型和延迟期间 abort 测试覆盖了否则会静默回归的边缘情况。

我的独立方案是"增加常量，也许分离预算"。PR 做到了这些加上历史推入顺序修复，后者我不会注意到。实现直白——无过度抽象，无投机性功能。

小提示：PR 正文仍写"217 个测试"但当前 head 已有 225 个。不是问题，只是文档滞后。

批准 ✅

— Qwen Code · qwen3.7-max

[qwen-code-ci-bot]

LGTM, looks ready to ship. ✅

[qwen-code-ci-bot]

No review findings. Downgraded from Approve to Comment: CI failing: Test (ubuntu-latest, Node 22.x).

— qwen3.7-max via Qwen Code /review

[yiliang114]

本轮处理了 continuation retry 路径的 Critical 问题：PROTOCOL_TAG_LEAK 现在和主发送循环一样使用 protocolTagLeakMaxRetries，日志中的 retry 分母也同步使用实际预算，并补了 output continuation 回归测试。

另外两条建议（budget exhausted 额外日志、把现有二分支 ternary 改成 switch 以防未来新增类型）属于诊断/泛化改进，不是当前红 CI 或 PR correctness 缺口；按本轮最小修复边界不扩大实现。

[qwen-code-ci-bot]

No issues found. LGTM! ✅

— qwen3.7-max via Qwen Code /review

[yiliang114]

本轮按 review comments 做了最小 closeout：

• 修复主发送循环里 invalid stream retry budget 共用同一个计数器的问题；transient invalid streams 和 PROTOCOL_TAG_LEAK 现在分别计数，避免 transient retry 预先耗尽 protocol-leak budget。
• 补充 mixed invalid-stream sequence 回归测试，覆盖 transient errors 后接 PROTOCOL_TAG_LEAK 仍可按自身 budget retry。
• 补充 invalid-stream retry delay 中途 AbortSignal 取消的覆盖，确认不会继续发起下一次 API 调用。
• continuation path 的 protocol-leak budget 评论未再改代码：当前 head 已经使用 maxContinuationRetries 并有 keeps protocol tag leak budget during output continuation 覆盖。

验证：cd packages/core && ../../node_modules/.bin/vitest run src/core/geminiChat.test.ts（224 passed），prettier --check src/core/geminiChat.ts src/core/geminiChat.test.ts，git diff --check。

[yiliang114]

本轮继续处理 14:51 自动 review 反馈：

• 修复 output continuation retry helper 仍共用一个 retryCount 的问题；continuation path 现在和主发送循环一样，transient invalid stream 与 PROTOCOL_TAG_LEAK 分别计数。
• delay/log/telemetry retry event 改为使用当前 error type 的 next retry count，避免 mixed error sequence 下 backoff 也被共享计数污染。
• 补充 continuation mixed-error 回归测试：两个 NO_FINISH_REASON 后接一个 PROTOCOL_TAG_LEAK，确认 protocol leak 仍能使用自己的 retry budget 并恢复成功。
• 未修改 ContentRetryFailureEvent.total_attempts 语义：当前字段表示本次 send 的总 API attempts；改为 per-type 或新增字段会扩大 telemetry contract，不放在这个 PR closeout 里。

验证：cd packages/core && ../../node_modules/.bin/vitest run src/core/geminiChat.test.ts（225 passed），prettier --check src/core/geminiChat.ts src/core/geminiChat.test.ts，git diff --check。

[qwen-code-ci-bot]

Reviewed — no blockers. Suggestions are inline.

[qwen-code-ci-bot]

[Suggestion] attempt_number in ContentRetryEvent is derived from the per-type counter (nextInvalidStreamRetryCount - 1), so it resets to 0 when switching error types. After 2 transient retries log attempt_number: 0 and attempt_number: 1, a subsequent protocol-tag-leak retry logs attempt_number: 0 again — the counter is non-monotonic within a single send operation.

At 3 AM, an engineer reading structured logs to trace a request lifecycle will see the counter go backwards and assume a bug. Consider renaming to error_type_attempt_number for clarity.

— qwen3.7-max via Qwen Code /review

[qwen-code-ci-bot]

[Suggestion] The continuation path has a protocol-tag-leak exhaustion test but no transient exhaustion test (4 retries with NO_FINISH_REASON then throw error). The throw error at this line when maxContinuationRetries === transientMaxRetries (4) is untested. Adding a continuation-path test analogous to the initial-path "should retry on invalid stream responses" test would close this gap.

— qwen3.7-max via Qwen Code /review

[wenshao]

✅ Local verification — reproduced & passing

I built and ran this PR locally in an isolated worktree at the PR head (dce63cc) as a merge reference. Bottom line: everything is green, and the new regression tests genuinely fail on pre-PR code — so they actually pin the intended behavior.

Environment: macOS (darwin 24.6.0) · Node v22.23.1 · fresh npm ci · packages/core built.

Results

Check	Result
geminiChat.test.ts (full affected file)	✅ 225 / 225 passed
tsc --noEmit (packages/core)	✅ 0 type errors
eslint on both changed files	✅ 0 findings

The 8 new/modified tests exercise specifically:

• Recovery on the 5th attempt after 4 consecutive empty (STOP, empty parts) streams, with backoff delays asserted at 2s / 4s / 6s / 8s, and history reduced to exactly [user, final model].
• Budget exhaustion after 5 total attempts → 4 ContentRetryEvents + 1 terminal ContentRetryFailureEvent (total_attempts: 5, final_error_type: NO_FINISH_REASON).
• PROTOCOL_TAG_LEAK budget unchanged at 2 (stops after 3 attempts), independent of the transient budget, in both the main generator and the output-continuation path.
• Abort during the retry delay surfaces immediately, with no further attempt.

Before / After — the tests really test the change

To confirm the new tests aren't vacuous, I overlaid main's pre-PR geminiChat.ts (the old maxRetries: 2) into the worktree while keeping this PR's test file. Three of the new tests then fail with exactly Model stream ended with empty response text. — the old 2-retry budget surfaces InvalidStreamError on the 3rd attempt. Restoring the PR source turns them green again. The PROTOCOL_TAG_LEAK-only test stays green in both states, confirming that budget is intentionally left untouched.

Notes for the merge decision

• The PR description mentions 217 tests; the current head has grown to 225 (it now also covers the PROTOCOL_TAG_LEAK vs transient budget separation). No concern — just a heads-up that the body text is slightly behind the branch.
• Matches the PR's own OS matrix: verified on macOS; Windows/Linux not run locally.
• Mechanism checks out: transientMaxRetries: 4 / protocolTagLeakMaxRetries: 2 are tracked as independent counters, and stream validation now throws before the partial assistant turn is pushed to history — so a failed attempt leaves no residue in the conversation.

中文说明

✅ 本地验证 —— 已复现并通过

作为合并参考，我在 PR head（dce63cc）上用独立 worktree 在本地构建并运行了本 PR。结论：全部通过，且新增回归测试在改动前的代码上会真实失败 —— 说明这些测试确实锁定了目标行为。

环境： macOS（darwin 24.6.0）· Node v22.23.1 · 全新 npm ci · 已构建 packages/core。

结果

检查项	结果
geminiChat.test.ts（受影响的完整测试文件）	✅ 225 / 225 通过
tsc --noEmit（packages/core）	✅ 0 类型错误
两个改动文件的 eslint	✅ 0 问题

8 个新增/修改的测试具体覆盖了：

• 连续 4 次空流（STOP + 空 parts）后在第 5 次尝试恢复成功，并断言退避延迟为 2s / 4s / 6s / 8s，历史记录精确收敛为 [user, 最终 model]。
• 预算耗尽时共 5 次尝试 → 4 个 ContentRetryEvent + 1 个终止的 ContentRetryFailureEvent（total_attempts: 5、final_error_type: NO_FINISH_REASON）。
• PROTOCOL_TAG_LEAK 预算保持为 2（第 3 次尝试后停止），与瞬态预算相互独立，在主生成器和输出续写（output continuation）两条路径上都成立。
• 重试延迟期间触发 abort 会立即抛出，不再进行下一次尝试。

修改前 / 修改后 —— 测试确实在测这个改动

为确认新测试不是「空测试」，我把 main 改动前的 geminiChat.ts（旧的 maxRetries: 2）覆盖进 worktree，同时保留本 PR 的测试文件。此时其中 3 个新测试会以完全一致的 Model stream ended with empty response text. 失败 —— 旧的 2 次重试预算会在第 3 次尝试时抛出 InvalidStreamError。恢复 PR 源码后重新变绿。而仅涉及 PROTOCOL_TAG_LEAK 的测试在两种状态下都保持绿色，印证该预算是有意保持不变的。

合并决策备注

• PR 描述里写的是 217 个测试，当前 head 已增长到 225（现在还覆盖了 PROTOCOL_TAG_LEAK 与瞬态预算的拆分）。无需担心，只是提醒正文略滞后于分支。
• 与 PR 自己的 OS 矩阵一致：已在 macOS 验证；Windows/Linux 未在本地运行。
• 机制无误：transientMaxRetries: 4 / protocolTagLeakMaxRetries: 2 作为相互独立的计数器跟踪；并且流校验现在会在把部分 assistant turn 推入历史之前抛错 —— 因此失败的尝试不会在对话中留下残留。

[wenshao]

@qwen-code /triage

[qwen-code-ci-bot]

LGTM, looks ready to ship. ✅