fix(core): auto-retry transient network errors during API calls - #7898
Conversation
…d API-call predicate The retry classification layer (classifyRetryError) already recognizes transport errors (ECONNRESET, ETIMEDOUT, etc.) as retryable, but neither defaultShouldRetry nor the inline shouldRetryOnError in geminiChat.ts consulted the classification for transport codes. TCP-level errors carry no HTTP status, so they fell through every predicate and propagated as raw [API Error: terminated (cause: read ECONNRESET)]. Wire the existing classifyRetryError transport detection into both retry decision points: 1. defaultShouldRetry in retry.ts — append a transport-kind check after the existing rate-limit / 5xx predicates, preserving all current behavior (including bounded retry for fail-fast quota 429s). 2. Inline shouldRetryOnError in geminiChat.ts makeApiCallAndProcessStream — add the same transport-kind check so the API-call-level retryWithBackoff wrapper covers network errors. The stream-level transport retry (gated on !streamYieldedChunk) already exists and is unchanged. Closes QwenLM#7831
|
✅ Qwen Triage finished — CI landed green on ✅ Qwen Triage 已完成 —— |
|
Thanks for the PR! Template: the headings differ from the template (Summary/Changes/Why this is safe vs What this PR does/Why it's needed/Reviewer Test Plan), but the content is all there — not worth a round-trip for heading names on a PR this size. Problem: observed bug with strong evidence. Issue #7831 documents five ECONNRESET occurrences with timestamps, durations clustering around 81–95s, and telemetry showing the pattern only appears at 150k+ token context. The retry-succeeds-immediately observation confirms this is transient. Clearly a real problem. Direction: squarely aligned — retrying transient transport errors is basic API resilience. The classification layer ( Size: 18 production lines (9 in geminiChat.ts, 9 in retry.ts) + 49 test lines. Well under any threshold. Not applicable. Approach: the scope feels exactly right. Two small additions wiring an existing, well-tested classifier into the two retry decision points that were missing it. No new abstractions, no new files, no drive-by changes. The stream-level transport retry already exists separately and is untouched. Moving on to code review. 🔍 中文说明感谢贡献! 模板:标题格式与模板不同(Summary/Changes/Why this is safe vs What this PR does/Why it's needed/Reviewer Test Plan),但内容完整——对于这么小的 PR,不值得为标题格式来回修改。 问题:已观测到的 bug,证据充分。Issue #7831 记录了五次 ECONNRESET,时间戳、持续时间集中在 81–95 秒,遥测数据显示仅在 150k+ token 上下文时出现。重试立即成功确认这是瞬态错误。确实是真实问题。 方向:完全对齐——重试瞬态传输错误是基本的 API 弹性。分类层( 规模:18 行生产代码(geminiChat.ts 9 行,retry.ts 9 行)+ 49 行测试代码。远低于任何阈值。不适用。 方案:范围恰好。两处小改动,将现有的、经过充分测试的分类器接入两个缺少它的重试判断点。没有新抽象、没有新文件、没有顺手改动。流级传输重试已经独立存在,未被触及。 进入代码审查 🔍 — Qwen Code · qwen3.8-max-preview Reviewed at |
Code ReviewIndependent proposal: given that Findings: no critical blockers, no convention violations. The The Both tests match the real error shapes from the issue (TypeError with nested cause for ECONNRESET, direct Error with code for ETIMEDOUT) and follow the existing test patterns (fake timers, attempt counting). One minor observation: TestingFinal CI results for
One row per check name (latest run); skipped checks omitted; failures sort first. / 每个检查名一行(取最新一次运行),省略 skipped,失败项排在最前。 The ubuntu unit test suite is still running. macOS and Windows tests were skipped (likely gated on ubuntu passing first). Precheck and classification checks passed. The author reports all 126 tests in retry.test.ts + retryErrorClassification.test.ts pass and typecheck is clean — noted as the author's claim, not independently verified here. Not verified: real-scenario ECONNRESET reproduction (requires a live API endpoint with the specific gateway timeout conditions from #7831). 中文说明代码审查独立方案: 鉴于 发现: 无关键阻塞问题,无规范违反。
两个测试匹配 issue 中的真实错误形状(ECONNRESET 的嵌套 cause TypeError,ETIMEDOUT 的直接 Error with code),并遵循现有测试模式。 测试Ubuntu 单元测试套件仍在运行。macOS 和 Windows 测试被跳过。预检和分类检查通过。作者报告 retry.test.ts + retryErrorClassification.test.ts 中全部 126 个测试通过,类型检查干净——记为作者声明,非独立验证。 未验证:真实场景 ECONNRESET 复现(需要具有 #7831 中特定网关超时条件的活跃 API 端点)。 — Qwen Code · qwen3.8-max-preview Reviewed at |
|
Confidence: 5/5 — clean across every stage; would merge without hesitation. This is exactly the kind of PR I like to see. The problem is real and well-documented (five ECONNRESET occurrences with telemetry in #7831), the root cause is clear (transport errors carry no HTTP status, so they fell through every retry predicate), and the fix is the minimum viable change: wire the existing classifier into the two places that weren't consulting it. Eighteen production lines, no new abstractions, no scope creep. My independent proposal matched the PR's approach exactly — I couldn't find a simpler path. The code reads well, the comments explain the "why" (transport errors carry no HTTP status), and the tests use the real error shapes from the issue. The existing behavior is fully preserved: rate-limit, 5xx, schema error, invalid argument, and 400 exclusions all still fire before the new transport check. The one thing I can't verify from here is the real-scenario behavior — reproducing the specific gateway timeout that triggers ECONNRESET requires the live Bailian endpoint under load. But the unit tests cover the retry mechanics, and the classification layer already has comprehensive transport-code coverage. The change is additive and bounded; the risk of regression is near zero. Approval deferred until CI lands green on 中文说明置信度:5/5 — 每个阶段都很干净;毫不犹豫可以合并。 这正是我喜欢看到的 PR。问题真实且记录充分(#7831 中有五次 ECONNRESET 及遥测数据),根因清晰(传输错误没有 HTTP 状态码,因此穿过了所有重试判断),修复是最小可行改动:将现有分类器接入两个未查询它的地方。十八行生产代码,没有新抽象,没有范围蔓延。 我的独立方案与 PR 的方法完全一致——找不到更简单的路径。代码可读性好,注释解释了"为什么"(传输错误没有 HTTP 状态码),测试使用了 issue 中的真实错误形状。现有行为完全保留:速率限制、5xx、schema 错误、无效参数和 400 排除都在新的传输检查之前触发。 唯一无法从这里验证的是真实场景行为——复现触发 ECONNRESET 的特定网关超时需要在负载下的活跃百炼端点。但单元测试覆盖了重试机制,分类层已有全面的传输错误码覆盖。改动是加法且有界的;回归风险接近零。 审批推迟到 CI 在 — Qwen Code · qwen3.8-max-preview Reviewed at |
|
@qwen-bot /takeover |
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship — CI landed green after the review. ✅
|
@qwen-code /takeover |
补充说明本 PR 只修改了 这意味着:
对于 #7831 的场景(150k+ context,~90s gateway timeout),如果 TTFT > 90s(大 context 下很可能),则首 token 前连接就被 reset,本 PR 的 API-call 层重试 + 已有的 stream transport retry 都能覆盖。如果 TTFT < 90s 但生成过程中被 reset,则仍会报错 — 这需要服务端调高 gateway timeout 来根治。 |
|
@qwen-code /takeover |
|
🤝 Takeover engaged: the autofix loop now manages this PR — it will address new review feedback and resolve base conflicts until the label is removed or the round cap is reached. Remove the 中文说明🤝 已接管:autofix 循环现在管理此 PR —— 将持续处理新的评审反馈与 base 冲突,直到移除标签或达到轮次上限。移除 |
|
🤖 Reviewed the latest feedback — no changes needed. Why, point by point: · 已审阅最新反馈——无需改动。逐点说明原因如下: Autofix review round — no action neededNo code change was made this round. There is no actionable feedback to address:
The PR remains at the approved commit 中文说明Autofix 审查轮次 — 无需操作本轮未做任何代码改动。没有需要处理的可操作反馈:
PR 仍停留在已批准的提交 Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。 🧠 Handled by Qwen Code · model/模型 |
Local build & real-network verification (head
|
| Failure phase | Real shape at the predicate | Classified | PR retries? |
|---|---|---|---|
| non-stream, RST mid-body (the #7831 string) | TypeError "terminated" → cause code=ECONNRESET (depth 1) |
transport |
YES ✅ |
| stream, RST after headers, before first event | same depth-1 shape | transport |
n/a — pre-existing stream layer handles it |
| any call, RST before headers (SDK-wrapped) | APIConnectionError → TypeError "fetch failed" → cause code=ECONNRESET (depth 2) |
unknown |
NO ❌ |
| raw undici fetch, RST before headers | TypeError "fetch failed" → cause code=ECONNRESET (depth 1) |
transport |
YES ✅ |
Review notes (non-blocking)
- The
geminiChat.tshunk cannot fire for OpenAI-SDK-wrapped connection failures. Pre-header failures reach the predicate asAPIConnectionErrorwith the socket code at cause-depth 2, butgetTransportCodeonly probeserror.codeanderror.cause.code— classification staysunknown(verified live: S2 identical on both builds). Today this is shielded by the SDK's 4 quick internal attempts, but a network blip lasting a few seconds exhausts those where the repo's 7-attempt backoff ladder would have ridden it out. Client-side timeouts similarly surface asAPIConnectionTimeoutError(no cause at all) and as a plainErroraftererrorHandler.ts's timeout rewrite (code-reading, not live-probed). Suggested follow-up: walk the cause chain (bounded, e.g. depth ≤ 4) ingetTransportCode. The hunk is still correct and reachable for depth-≤1 establishment shapes (raw-fetch/undici providers), so it should stay. - No test pins the
geminiChat.tsinline predicate — the two new unit tests exercisedefaultShouldRetryonly. A predicate-level test would come naturally with the note-1 follow-up. - Landscape: complements open fix(core): retry mid-stream transport failures as continuations #7876 (continuation retry after chunks have been yielded — the remaining uncovered phase); both touch
geminiChat.ts, so whichever merges second needs a trivial rebase. MCP-client and weixin-channelretryWithBackoffcall sites keep their own predicates — unchanged, consistent with the PR's stated scope. All other LLM call sites usedefaultShouldRetryand get the fix automatically.
中文版本(完整验证报告)
本地构建与真实网络验证(head e6122b3)
结论:✅ 可合并。 核心主张——瞬态网络错误在 API 调用层自动重试——已在生产非流式路径上端到端验证:真实 TCP RST 服务器 + 真实 OpenAI SDK + 仓库真实的 retryWithBackoff。下附两条不阻塞合并的跟进建议。
环境:macOS (darwin 24.6)、Node v22.23.1、隔离 worktree(e6122b3,merge-base 6a432ad)、全新 npm ci;每次 A/B 前均以标记 grep 验证 base/PR 构建产物。
1. 单测、lint、typecheck 全绿
PR head 上 retry.test.ts 89/89、geminiChat.test.ts 259/259;三个改动文件 eslint 与 tsc --noEmit 干净。
2. 源码交换 A/B——新测试具有判别力
用 merge-base 的 retry.ts + geminiChat.ts 覆盖源码、保留 PR 的测试文件:恰好两个新测试失败(ECONNRESET、ETIMEDOUT);87 个既有测试两侧均通过——限流 / 5xx / fast-fail 路径行为无变化。
3. 真实网络差分——修复在线上生效
本地 HTTP 服务器第 1 次请求发送 200 + 部分 JSON 响应体后 socket.resetAndDestroy()(真实 TCP RST),第 ≥2 次正常返回。真实 OpenAI SDK(禁用其内部重试)配合各构建的 retryWithBackoff 生产默认参数——与 client.ts:3220 / baseLlmClient.ts:289,440 包装所有非流式 LLM 调用(JSON 侧查询、goal 评审、压缩/摘要)的方式完全一致:
- base 构建:1 次请求即失败,44ms——
TypeError: terminated (cause: read ECONNRESET),与 Repeated ECONNRESET on streaming responses when context exceeds ~150k tokens #7831 报告的原始错误字符串完全一致 - PR 构建:约 1.5 秒指数退避后第 2 次请求成功
4. 编译后 CLI 真机 E2E(隔离 $HOME,本地 SSE mock)
- S1——SSE 头已发、首事件前 RST(仅 1 次): base 与 PR 构建都自动恢复并输出标记——这是既有的流层 transport 重试(fix(core): auto-retry transport stream errors before the first chunk #5171),PR 正确地未改动该层。
- S2——每次请求都在响应头前 RST: 两个构建以相同方式失败(
[API Error: Connection error. (cause: UND_ERR_SOCKET: other side closed)],服务端观测到 4 次连接)——这 4 次是 OpenAI SDK 自身的内部重试(maxRetries = 3),仓库层退避从未启动。见注 1。
5. 错误形状探针——真实捕获错误 → PR 的真实分类器
每行都是 SDK/undici 对真实 RST 服务器抛出的真实错误对象,原样喂给编译后的 classifyRetryError:
| 失败阶段 | 到达判定函数的真实形状 | 分类结果 | PR 是否重试 |
|---|---|---|---|
| 非流式、响应体中途 RST(#7831 的错误串) | TypeError "terminated" → cause code=ECONNRESET(深度 1) |
transport |
是 ✅ |
| 流式、头已发、首事件前 RST | 同上(深度 1) | transport |
不适用——既有流层处理 |
| 任意调用、响应头前 RST(SDK 包装) | APIConnectionError → TypeError "fetch failed" → cause code=ECONNRESET(深度 2) |
unknown |
否 ❌ |
| 原生 undici fetch、响应头前 RST | TypeError "fetch failed" → cause code=ECONNRESET(深度 1) |
transport |
是 ✅ |
审阅备注(不阻塞合并)
geminiChat.ts改动点对 OpenAI-SDK 包装的连接失败无法生效。 响应头之前的失败以APIConnectionError形式到达判定函数,socket code 位于 cause 链深度 2,而getTransportCode只探测error.code与error.cause.code——分类停在unknown(真机验证:S2 两构建行为一致)。目前该场景由 SDK 的 4 次快速内部重试兜底,但持续数秒的网络抖动会耗尽这些快速重试,而仓库层 7 次指数退避本可扛过去。客户端超时同理:APIConnectionTimeoutError无 cause,且errorHandler.ts的超时改写会替换为不带 code 的普通Error(此点为代码走读,未做真机探针)。**建议跟进:**在getTransportCode中有界遍历 cause 链(如深度 ≤ 4)。该改动点对深度 ≤1 的建立期错误形状(原生fetch/undici 的 provider)仍然正确且可达,应予保留。- 没有测试钉住
geminiChat.ts的内联判定改动——两个新单测只覆盖defaultShouldRetry。随注 1 的跟进自然可补上判定级测试。 - **全景:**与开放中的 fix(core): retry mid-stream transport failures as continuations #7876 互补(已产出 chunk 后的续写式重试——剩余未覆盖阶段);两者都改
geminiChat.ts,后合入者需做一次轻量 rebase。MCP client 与 weixin channel 的retryWithBackoff调用点保留各自判定——未改动,符合 PR 声明的范围。其余 LLM 调用点均走defaultShouldRetry,自动获得修复。
|
🤖 Addressed the latest review feedback (round 1/100). What changed, and what I pushed back on: · 已处理最新评审反馈(第 1/100 轮)。改动内容与我反驳保留之处如下: Review feedback addressedThanks for the exhaustive live verification and the precise error-shape probe. The two actionable notes are implemented; the landscape note needs no code change. Note 1 — walk the cause chain in
|
|
Recovered after restart; restarting task |
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
中文说明
— qwen3.7-max via Qwen Code /review
|
🤖 Reviewed the latest feedback — no changes needed. Why, point by point: · 已审阅最新反馈——无需改动。逐点说明原因如下: Autofix review round — no action neededThis round's feedback contains no actionable findings, so no code change was made.
The PR stands as-is. No commit was created this round. 中文说明Autofix 审查轮次 — 无需处理本轮反馈不包含任何可处理的发现,因此未做任何代码改动。
本 PR 保持现状。本轮未创建任何提交。 Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。 🧠 Handled by Qwen Code · model/模型 |
|
Released in v0.21.1. |




What this PR does
Adds transient network errors (ECONNRESET, ETIMEDOUT, ECONNREFUSED, etc.) to the set of conditions that trigger automatic retry at the API-call level. Previously, these TCP-level errors carried no HTTP status code and fell through every existing retry predicate, surfacing as raw
[API Error: terminated (cause: read ECONNRESET)]to the user. The fix wires the existing transport-error classification into both retry decision points so that network glitches are retried with exponential backoff, just like rate-limit and server errors already are.Why it's needed
Closes #7831.
Users with large contexts (150k+ tokens) experience repeated
ECONNRESETfailures when a server-side gateway timeout (~90s) kills the TCP connection before the model finishes streaming. The next manual retry always succeeds immediately, confirming these are transient. The client already classifies these errors as transport-retryable and already retries them mid-stream (when no chunk has been yielded), but the API-call-level retry path did not consult that classification — so the error propagated to the user instead of being retried automatically.Reviewer Test Plan
How to verify
cd packages/core && npx vitest run src/utils/retry.test.ts— confirm the two new tests ("should retry on transient network errors (ECONNRESET) by default" and "should retry on ETIMEDOUT by default") pass alongside all existing tests.Evidence (Before & After)
N/A (non-UI change; retry behavior is internal)
Tested on
Environment (optional)
Unit tests only (
npx vitest run).Risk & Scope
Linked Issues
Closes #7831
中文说明
本 PR 做了什么
将瞬态网络错误(ECONNRESET、ETIMEDOUT、ECONNREFUSED 等)加入 API 调用级别的自动重试条件。此前,这些 TCP 层错误不携带 HTTP 状态码,会穿透所有现有的重试判定逻辑,以原始的
[API Error: terminated (cause: read ECONNRESET)]形式直接暴露给用户。本次修复将已有的传输层错误分类接入两个重试判定点,使网络抖动像限流和服务端错误一样自动进行指数退避重试。为什么需要
关闭 #7831。
大上下文(150k+ tokens)用户在服务端网关超时(约 90 秒)切断 TCP 连接时,会反复遇到
ECONNRESET失败。每次手动重试都立即成功,证实这些是瞬态错误。客户端已将这些错误归类为可重试的传输层错误,且已在流式传输中途(未产出任何 chunk 时)进行重试,但 API 调用级别的重试路径未引用该分类——因此错误直接传播给用户,而非自动重试。审阅者测试计划
如何验证
cd packages/core && npx vitest run src/utils/retry.test.ts——确认两个新测试("should retry on transient network errors (ECONNRESET) by default" 和 "should retry on ETIMEDOUT by default")与所有现有测试一起通过。证据(前后对比)
N/A(非 UI 变更;重试行为为内部逻辑)
测试环境
环境(可选)
仅单元测试(
npx vitest run)。风险与范围
关联 Issue
关闭 #7831