fix(integration): make interactive read-then-write test deterministic (#8060) - #8064
Merged
Conversation
…#8060) Drive the multi-turn conversation with the fake OpenAI server instead of a live LLM. The real model made this test flaky on main: it could choose different tools, phrase the read result without the literal version, or settle the first turn on its own schedule. Scripting the exact read-then-write turns keeps the interactive mechanics (typed input, tool execution, file mutation) under test while removing the nondeterminism, matching the fake-server pattern already used by the other multi-turn interactive tests.
Collaborator
Author
E2E Report — issue #8060Failure under testThe main-branch Root causeThe test's nondeterminism came from depending on a real model for a multi-turn interactive flow. A live model can:
The prior fixes only lengthened waits; they never removed the nondeterminism, which is why the test kept flaking. The other multi-turn interactive tests in this repo ( FixConverted the test to drive the conversation with the fake OpenAI server, scripting the exact turn sequence: How to verifyBuild the bundle and run the single integration test repeatedly; it should pass deterministically and quickly (~12s) with no live model: npm run build && npm run bundle
cd integration-tests && QWEN_SANDBOX=false npx vitest run interactive/file-system-interactive.test.tsObserved: the test passed on 4 consecutive runs (1 initial + 3 stability reruns), each ~12s, with the fake server receiving the scripted four requests and the file ending as Verification
Note: the original failure is a CI/timing-specific flake against a live model; the exact CI E2E environment is not reproducible on this runner. The fix removes the live-model nondeterminism that caused the flake, and the converted test is verified locally against the bundled CLI as the surrogate. The workflow's independent CI remains the final gate. 中文说明E2E 报告 — issue #8060被测的失败main 分支的 根因测试的非确定性来自：在一个多轮交互流程中依赖真实模型。真实模型可能：
此前的修复只是加长了等待时间，从未移除非确定性，因此测试持续 flake。本仓库中其他多轮交互测试（ 修复将测试改为用 fake OpenAI server 驱动对话，脚本化精确的轮次序列： 如何验证构建 bundle 并反复运行该单个集成测试；它应当确定性地、快速地（约 12 秒）通过，且不依赖真实模型： npm run build && npm run bundle
cd integration-tests && QWEN_SANDBOX=false npx vitest run interactive/file-system-interactive.test.ts观察结果：测试连续 4 次通过（1 次初始 + 3 次稳定性重跑），每次约 12 秒，fake server 收到脚本化的四个请求，文件最终内容为 验证
说明：原始失败是针对真实模型的 CI/时序相关 flake；本 runner 无法复现确切的 CI E2E 环境。该修复移除了导致 flake 的真实模型非确定性，转换后的测试在本地针对打包后的 CLI 作为替代验证。工作流的独立 CI 仍是最终门禁。 🧠 Handled by Qwen Code · model/模型 |
Collaborator
Collaborator
|
Thanks for the PR! Template looks good ✓ Problem: observed, not theoretical. #8060 is a real main-branch Direction: aligned. Making a flaky multi-turn interactive test deterministic by driving it with the repo's existing fake OpenAI server is squarely test-infra hygiene, and it brings this test in line with the sibling interactive tests that already use that pattern. CHANGELOG: no direct reference (test-only change); the area is relevant. Size: not applicable — no core paths touched. Single test file, +133/−45, 0 production logic lines. Approach: the scope feels right and notably minimal. It reuses Risk: no elevated risk signals — no high-risk production paths matched. Moving on to code review. 🔍 中文说明感谢贡献！ 模板完整 ✓ 问题：已观测到，非理论性。#8060 是一次真实的 main 分支 方向：对齐。用仓库现有的 fake OpenAI server 驱动这个易 flake 的多轮交互测试使其确定化，属于纯粹的测试基础设施改进，并让该测试与已采用同一模式的同类交互测试保持一致。CHANGELOG：无直接引用（仅测试改动）；该领域相关。 规模：不适用——未触及核心路径。单个测试文件，+133/−45，0 生产逻辑行。 方案：范围恰当且相当精简。复用 风险：无升级风险信号——未匹配高风险生产路径。 进入代码审查 🔍 — Qwen Code · qwen3.8-max-preview Reviewed at |
Collaborator
Code reviewI read the PR title + "Why it's needed" first and proposed my own fix before looking at the diff: drive the two-turn conversation with the repo's deterministic fake OpenAI server (the same pattern The implementation is clean and idiomatic:
I checked the request/response indexing against Test evidenceCI on
One honest caveat the maintainer should know: PR CI did not execute the modified test. The green Sandboxed verification would settle this: Real-scenario testing: N/A — test-only change with no user-visible/TUI behavior change (and this is an unattended CI run). 中文说明代码审查我先只看 PR 标题和"为什么需要"，独立提出自己的方案：用仓库确定性的 fake OpenAI server 驱动这个两轮对话（与 实现干净且符合惯例：
我对照 测试证据
一个维护者应当了解的诚实说明：PR CI 并未执行被修改的测试。 绿色的 沙箱验证可以坐实这一点： 真实场景测试：N/A——仅测试改动，无用户可见/TUI 行为变化（且本次为无人值守 CI 运行）。 — Qwen Code · qwen3.8-max-preview Reviewed at |
github-actions
Bot
added
the
review/self-reported
Collaborator
|
Confidence: 4/5 — clean, idiomatic, test-only fix for a real observed flake; the only nit is that PR CI never runs the interactive suite, so the converted test's pass rests on the author's local runs plus a strong by-construction argument. Stepping back: this is exactly the kind of PR the gate should let through. The problem is real and observed (#8060 — this exact test failing on The one reservation, and it's non-blocking: because the interactive suite is wired to Approving, pinned to the reviewed commit. 中文说明置信度：4/5 —— 干净、符合惯例、仅涉及测试的修复，针对一个真实观测到的 flake；唯一的小问题是 PR CI 从不运行交互式套件，因此转换后测试的通过依赖于作者的本地运行加上一个有力的"按构造"论证。 退一步看：这正是 gate 应当放行的那类 PR。问题真实且已观测（#8060——正是这个测试在 唯一的保留意见，且非阻塞：由于交互式套件接入 批准，锁定在被审查的 commit 上。 — Qwen Code · qwen3.8-max-preview Reviewed at |
Collaborator
Author
|
🔀 Base updated: red check(s) [Test (ubuntu-latest, Node 22.x)] pass on current main — merged current main via update-branch; CI will re-run. 中文说明🔀 已更新 base：红色检查 [Test (ubuntu-latest, Node 22.x)] 在当前 main 上通过 —— 已通过 update-branch 合入当前 main，CI 将重新运行。 |
Contributor
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed. Suggestions are inline. Not reviewed: build-and-test — Integration Tests (CLI, No Sandbox) was skipped in CI and its suite did not run locally.
— qwen3.8-max-preview via Qwen Code /review
Collaborator
Author
doudouOUC
previously approved these changes
Jul 29, 2026
doudouOUC
left a comment
doudouOUC
left a comment
Collaborator
There was a problem hiding this comment.
No issues found. LGTM! ✅
中文说明
未发现问题。LGTM！✅
— qwen3.7-max via Qwen Code /review
Collaborator
Author
|
🤖 Addressed the latest review feedback (round 1/10). What changed, and what I pushed back on: · 已处理最新评审反馈（第 1/10 轮）。改动内容与我反驳保留之处如下： Review feedback — round summaryInline findings
ChangesAdded Conflict notesNo conflicts ( Verification
中文说明审查反馈 — 本轮摘要行内发现
变更内容在 冲突说明无冲突（ 验证
Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。 Re-review when you have a moment. After round 10 this bot stops and leaves the PR for a human. · 有空请复审；第 10 轮后本 bot 停止并将 PR 交给人工。 🧠 Handled by Qwen Code · model/模型 |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed. Suggestions are inline.
中文说明
已审查。 建议见行内评论。
— qwen3.8-max-preview via Qwen Code /review
Collaborator
Author
|
🤖 Reviewed the latest feedback — no changes needed. Why, point by point: · 已审阅最新反馈——无需改动。逐点说明原因如下： No action taken — transient CI failureThe only feedback this round is a failed check: Test (ubuntu-latest, Node 22.x). Evidence this is a transient infrastructure failure, not a code defect
RecommendationRe-run the CI check. No code change is needed. 中文说明未采取行动 — CI 瞬时故障本轮唯一的反馈是一个失败的检查：Test (ubuntu-latest, Node 22.x)。 证据表明这是瞬时基础设施故障，而非代码缺陷
建议重新运行 CI 检查。无需修改代码。 Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。 🧠 Handled by Qwen Code · model/模型 |
Collaborator
Author
|
🤖 Addressed the latest review feedback (round 2/10). What changed, and what I pushed back on: · 已处理最新评审反馈（第 2/10 轮）。改动内容与我反驳保留之处如下： Review feedback addressedBoth inline suggestions from the automated reviewer are implemented in [rc:3677378675]
|
qwen-code-ci-bot
previously approved these changes
Jul 29, 2026
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
No issues found. LGTM! ✅
中文说明
未发现问题。LGTM！✅
— qwen3.8-max-preview via Qwen Code /review
Collaborator
Verification report — built and ran this locallyVerdict: LGTM. The change does what it claims: the test is deterministic, it still has teeth on all four preserved assertions, and it passes in the Setup. Fresh worktree at PR head 1. Determinism, contention, container lane, live baseline
raw text (if the image has not resolved yet)
2. Do the four preserved assertions still have teeth?I re-ran the PR's own test body eight times, perturbing exactly one thing per run. If an assertion cannot be made to fail, it is decoration.
raw text (if the image has not resolved yet)
A3's result is the one I care about most: it means the test still genuinely exercises the interactive multi-turn mechanics, which is the unique value the PR says it is preserving. A4's result means the disk assertion is not satisfiable by a tool call that merely claims success. 3.
|
| 断言 | 我怎么尝试破坏它 | 结果 |
|---|---|---|
A1 read_file 工具调用 |
index 0 改成脚本化 read_many_files |
变红 —— 有牙齿 |
A2 输出中出现 '1.0.0' |
从脚本化回复里去掉 1.0.0 |
变红 —— 有牙齿 |
A3 write_file/edit 工具调用 |
干脆不键入第二个 prompt | 变红 —— 键入的 PTY 输入确实是承重的 |
A4 文件内容含 1.0.1 |
脚本化 write_file 写到另一个路径 |
变红 —— 锚定在真实文件系统上，而不是工具日志 |
其中 A3 是我最在意的一条：它说明这个测试仍然真正在验证交互式多轮机制 —— 也就是 PR 声称要保留的那份独有价值。A4 则说明磁盘断言无法被「只是声称成功」的工具调用满足。
3. ui.enableFollowupSuggestions: false 是承重的（这个加得好）
单独去掉这一个设置，测试就在 A3 变红。逐请求审计显示了原因：一个 [SUGGESTION MODE: Suggest what the user might naturally type next.] 请求落在了 requestIndex 2，占掉了脚本留给 write_file 的位置，于是写入那一轮拿到的是收尾的 content 回复而不是工具调用。这正是 PR 在 toHaveLength(4) 那条断言旁边注释里预测的偏移，而那条断言就是用来暴露它的。顺带一提：同类的 interactive/protocol-tags-interactive.test.ts 并没有设这一项 —— 它在本机仍然通过（5.3 s），因为它只有单轮；但对任何以后想给它加第二轮的人来说，这是个潜在坑。
相比之下，memory.enableManagedAutoMemory/Dream: false 在这个场景里是空转的 —— 加不加，server 都恰好收到 4 个请求。无害，也和同类测试保持一致；只是说明它们属于保险起见，而不是修复的一部分。
4. main 上实际失败的是什么 —— 而它不在列出的三条根因里
我拉了被引用那次运行的日志（run 30468059228，E2E Test (Linux) - sandbox:docker - shard 2/3，main @ eb5798b7f5d4），并把最后一帧 TUI 渲染了出来（见上方英文部分第三张图）。
第二轮确实已经提交，模型也确实理解了（The user wants me to update the version in version.txt from 1.0.0 to 1.0.1）。然后它吐了 3 个 token 就卡住了 —— spinner 停在 3s · ↑ 3 tokens · esc to cancel，直到轮询超时。所以实际观察到的模式是第二轮流式响应中途卡死，既不是工具选择、也不是措辞、也不是第一轮收尾太晚。
我把模型分别钉死到 PR 列出的三条根因上，对基线（main）测试主体逐字节复现：
- 「用不含字面量
1.0.0的措辞给出读取结果」 → 能复现为硬失败（A2 变红）。✅ - 「选择不同的工具」 → 能复现为硬失败（A1 变红）。✅
- 「按自己的节奏结束第一轮，从而让第二次键入的 prompt 与之竞争」 → 没能复现。 我让第一轮在
1.0.0已经渲染之后继续做工具轮次、每轮延迟 3 s，使第二个 prompt 被键入到一个仍在进行中的轮次里。CLI 把它排队了，等第一轮结束后照常执行，测试依旧通过。所以这条假设在我的测试里得不到支持。
这些都不改变结论 —— 脚本化的本地 server 不可能卡死，所以修复同样覆盖了真实那条模式。只是描述文字略有偏差；建议改成引用「中途卡死」，因为「模型可能在流式过程中挂住」比「它可能选到别的工具」是更强的去掉真实模型的理由。
5. 两件小事
- A2 并不观察读取结果 —— PR 自己新加的注释已经说了这一点，我也确认了：磁盘上放
9.9.9、脚本化回复仍说1.0.0，测试依旧是绿的（上表fixture-999）。真实 CI 那一帧也印证了机制：TUI 把成功的读取渲染成一行✓ Read version.txt，而1.0.0来自模型自己的话◆ The version is 1.0.0.。所以这不是本 PR 引入的退化 —— 在main上这条断言同样依赖模型措辞。如果希望 A2 真正有意义，加一行就够了，因为工具结果就在 transcript 里：expect(JSON.stringify(fakeServer.requests[1]!.body)).toContain('1.0.0');
- 顺手修掉的 PTY 泄漏，描述里没提 —— 基线测试从不 kill 子进程（
const { ptyProcess } = rig.runInteractive()，没有finally），所以main上每跑一次就漏一个 CLI 进程。PR 的finally { ptyProcess.kill(); await promise; }修掉了它，值得在描述里写一句。
6. 改动文件的 presubmit
prettier --check 干净、eslint 干净，且归因到 integration-tests/interactive/file-system-interactive.test.ts 的 TS 报错为 0。（附带记录：integration-tests/tsconfig.json 本身都解析不了 —— paths 里那个 "//" 注释键会触发 TS5063 并让 tsc -p 直接中止。这是既有问题，与本 PR 无关。）
复现命令见上方英文部分的 Repro 段。给后续重复此验证的人一个提示：fake server 的 requestIndex 是每个 server 的全局计数器，任何会发起后台模型调用的东西（followup 建议、managed-memory 抽取）都会把脚本顺序挪位；如果这个测试以后要加第三轮，按 transcript 内容路由会比按 requestIndex 更稳。
Address review feedback: assert the read_file tool result the CLI sent back (requests[1]) contains '1.0.0', so the version check observes the real read result rather than only the fake model's scripted echo. Also correct the flake rationale comment to cite the verified mid-stream stall on turn 2 instead of an unsupported late-settle hypothesis.
Collaborator
Author
|
🤖 Addressed the latest review feedback (round 3/10). What changed, and what I pushed back on: · 已处理最新评审反馈（第 3/10 轮）。改动内容与我反驳保留之处如下： Autofix review round — PR #8064 (issue #8060)Thanks for the thorough verification report — LGTM received. Two of the Feedback → decision
The remaining observations are informational and need no change: the No inline review comments were present this round, so there are no review Verification
中文说明Autofix 审查轮次 —— PR #8064（issue #8060）感谢这份详尽的验证报告 —— 已收到 LGTM。其中两条不阻塞合并的说明可以在代码中 反馈 → 处理
其余观察都属于信息性、无需改动： 本轮没有任何行内审查评论，因此没有需要解决或回复的审查线程。也无需解决冲突 验证
Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。 Re-review when you have a moment. After round 10 this bot stops and leaves the PR for a human. · 有空请复审；第 10 轮后本 bot 停止并将 PR 交给人工。 🧠 Handled by Qwen Code · model/模型 |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed. Not reviewed: build-and-test — Integration Tests (CLI, No Sandbox) was skipped in CI and its suite did not run locally.
— qwen3.8-max-preview via Qwen Code /review
Collaborator
Author
|
🤖 Reviewed the latest feedback — no changes needed. Why, point by point: · 已审阅最新反馈——无需改动。逐点说明原因如下： No action requiredThe only feedback this round is an informational COMMENTED review from the automated reviewer noting that the "Integration Tests (CLI, No Sandbox)" CI job was skipped and the suite did not run locally. This is an observation about CI configuration, not a defect report or a change request. There are no inline comments, no failed checks, no still-red checks, and no CHANGES_REQUESTED items to address. The PR diff (deterministic fake-model rewrite of the interactive file-system test) is unchanged. 中文说明无需操作本轮唯一的反馈是自动审查机器人的一条信息性 COMMENTED 评审，指出 "Integration Tests (CLI, No Sandbox)" CI 作业被跳过且该套件未在本地运行。这是对 CI 配置的观察说明，而非缺陷报告或变更请求。 没有行内评论、没有失败的检查、没有持续红色的检查、也没有 CHANGES_REQUESTED 项目需要处理。PR 的差异（将交互式文件系统测试重写为确定性假模型驱动）保持不变。 Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。 🧠 Handled by Qwen Code · model/模型 |
wenshao
approved these changes
Jul 30, 2026
Collaborator
|
@qwen-code /triage |
qwen-code-ci-bot
approved these changes
Jul 30, 2026
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
LGTM, looks ready to ship. ✅
Collaborator
|
Released in v0.21.2. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
This PR makes the interactive read-then-write E2E test deterministic. The test previously drove a two-turn interactive session — read
version.txt, then change the version to1.0.1— against a live LLM. It now drives the same conversation with the repository's deterministic fake OpenAI server, scripting the exact turn sequence: aread_filetool call, an assistant reply containing1.0.0, awrite_filetool call writing1.0.1, and a closing assistant reply. All four existing behavior assertions are preserved unchanged, so the interactive mechanics still under test are the typed PTY input, the tool execution, and the file mutation. Background model requests that could shift the scripted request order are suppressed through settings, and the container-sandbox networking options are applied the same way the sibling fake-server tests do it.Why it's needed
This test failed on
mainand has a long history of intermittent flakes (it was hardened in #7105, #7113, and #7943 without the flaking stopping). The root cause is that a live model is nondeterministic: it can pick a different tool than the waits expect, phrase the read result without the literal1.0.0the test polls for, or settle the first turn on its own schedule so the second typed prompt races it. Lengthening waits never removed that nondeterminism. The other multi-turn interactive tests in this repo already avoid this exact problem by using the fake OpenAI server, and this change brings this test in line with that established pattern.Reviewer Test Plan
How to verify
Build the bundle and run the single integration test a few times; it should pass deterministically and quickly (about 12 seconds) with no live model involved:
Confirm that the test passes on repeated runs, that the fake server receives the four scripted requests in order, and that
version.txtends as1.0.1. Also confirm the assertions still guard the intended behavior: aread_filecall is observed,1.0.0appears in the rendered output, awrite_file/editcall is observed, and the file content contains1.0.1.Evidence (Before & After)
N/A — this is a test-only change with no user-visible/TUI behavior change. Before: the test depended on a live LLM and flaked intermittently on
main. After: the test is driven by a scripted fake model and passed 4 consecutive local runs (~12s each) against the bundled CLI.Tested on
Environment (optional)
Local integration run against the bundled CLI (
npm run build && npm run bundle) withQWEN_SANDBOX=false. The test isskipIf(win32), unchanged from before.Risk & Scope
integration-tests/cli/file-system.test.ts > should perform a read-then-write sequence), so the unique value here — the interactive multi-turn mechanics — is preserved while the flake source is removed.Linked Issues
Fixes #8060
中文说明
本 PR 做了什么
本 PR 让交互式“先读后写”E2E 测试变为确定性。该测试此前针对真实 LLM 驱动一个两轮交互会话——读取
version.txt，再把版本改为1.0.1。现在改用本仓库确定性的 fake OpenAI server 驱动同样的对话，脚本化精确的轮次序列：一次read_file工具调用、一条包含1.0.0的助手回复、一次写入1.0.1的write_file工具调用，以及一条收尾的助手回复。四个既有行为断言全部原样保留，因此被测的交互机制仍然是 PTY 键入输入、工具执行和文件写入。可能使脚本化请求顺序偏移的后台模型请求通过设置项关闭，容器沙箱的网络选项也以与同类 fake-server 测试相同的方式应用。为什么需要
该测试在
main上失败，并有长期偶发 flake 的历史（曾在 #7105、#7113、#7943 中加固，但 flake 仍未停止）。根因在于真实模型是非确定性的：它可能选择与等待逻辑预期不同的工具、用不含字面量1.0.0的措辞给出读取结果（而测试正是轮询该字面量），或按自己的节奏结束第一轮从而让第二次键入的 prompt 与之竞争。加长等待时间从未移除这种非确定性。本仓库中其他多轮交互测试已经通过使用 fake OpenAI server 规避了完全相同的问题，本改动让该测试与这一既定模式保持一致。评审者测试计划
如何验证
构建 bundle 并多次运行该单个集成测试；它应当确定性地、快速地（约 12 秒）通过，且不涉及真实模型：
确认测试在重复运行下通过、fake server 按序收到四个脚本化请求、且
version.txt最终为1.0.1。同时确认这些断言仍然守护预期行为：观察到read_file调用、渲染输出中出现1.0.0、观察到write_file/edit调用、文件内容包含1.0.1。证据（前后对比）
N/A —— 这是仅涉及测试的改动，没有用户可见/TUI 行为变化。改动前：测试依赖真实 LLM，在
main上偶发 flake。改动后：测试由脚本化的 fake 模型驱动，针对打包后的 CLI 连续 4 次本地通过（每次约 12 秒）。测试环境
环境（可选）
针对打包后的 CLI（
npm run build && npm run bundle）在QWEN_SANDBOX=false下进行本地集成运行。该测试为skipIf(win32)，与之前一致。风险与范围
integration-tests/cli/file-system.test.ts > should perform a read-then-write sequence），因此此处独有的价值——交互式多轮机制——得以保留，同时移除了 flake 来源。关联 Issue
Fixes #8060