fix(test): isolate integration tests from the host global qwen dir - #10337
Merged
Conversation
The integration suites spawn the real CLI without isolating the global qwen dir, so whatever sits in the host's `~/.qwen` shapes the run. Hosted runners have an empty one and never noticed; the persistent pool does not, and there the saved memories left by earlier jobs made managed auto-memory recall issue its own model request ahead of the agent's first turn. The SDK suites script their fake model server by request index, so that extra request shifted every index: each scripted tool call landed on the recall selector and the turn under test got the trailing text instead, failing 42 cases across permission control and tool control on both Linux legs while macOS and the other shards stayed green. The same reds reproduce on any developer machine that has saved memories. Give each run its own global qwen dir, carrying the host's configuration across but none of the files that dir accumulates. Configuration has to come along whole: the suites that talk to a real model rely on ambient auth, which can live in the credentials block, in the environment block, or in the model routing, and the persistent pool's own credential source is not knowable from here. The accumulated files - saved memories, tool-usage history, extensions, skills, commands - are what a run has no business depending on, and dropping them is what makes the suites deterministic again.
Collaborator
|
|
Collaborator
|
Thanks for the PR! Template looks good ✓ Problem: observed, not theoretical — this is the repair for #10330, and the failing E2E run is linked there. The root-cause chain is concrete: a populated Direction: aligned — test-harness hermeticity, not product behavior. It reuses the documented Size: not applicable — confined to Approach: scope feels right, and carrying the host configuration across whole (settings.json + oauth_creds.json) instead of filtering it is well argued — ambient auth can live in the credentials block, the env block, or model routing, and there is no safe subset. Callers that pin Risk: no elevated risk signals — nothing matches the revert-correlated paths. Moving on to code review. 🔍 中文说明感谢贡献！ 模板完整 ✓ 问题：是已观测到的问题，不是理论推演——这是对 #10330 的修复，失败的 E2E run 就链接在那里。根因链条很具体：持久化 runner 池上已填充的 方向：对齐——这是测试框架的密封性问题，不涉及产品行为。它复用了文档化的 规模：不适用——改动仅限 方案：范围合理。把宿主配置整体带过去（settings.json + oauth_creds.json）而不是做过滤，论证是成立的——ambient auth 可能放在 credentials 块、env 块或 model routing 里，不存在一个可以单独安全携带的子集。自己指定 风险：无升级风险信号——未命中与 revert 相关的高风险路径。 进入代码审查。🔍 — Qwen Code · qwen3.8-max Reviewed at |
Collaborator
Code reviewRead against the base: my independent take on this problem was exactly what this PR does — redirect One process note first: What checks out:
What I'd ask about (none of these block):
TestingThis was an unattended run, so no PR code was built or executed here; the evidence below is the PR's own CI, read via the API at the reviewed commit. Final CI results for
One row per check name (latest run); skipped checks omitted; failures sort first. / 每个检查名一行（取最新一次运行），省略 skipped，失败项排在最前。 Two things worth knowing about what this CI can and cannot prove:
Not verified: the polluted-host before/after. The 42-failed → 52-passed and 152-passed tables in the PR body are the author's local runs, quoted here as the author's claim, not independently re-run. They match the failing run's signature exactly (same suites, same count, both Linux legs). Sandboxed verification would settle the remaining gap: 中文说明代码审查先说结论：如果不看 diff，我自己对这个问题给出的方案也正是这个 PR 的做法——在 先说明一个过程细节：审查进行中分支合入了 核对无误的部分：
想追问的点（均不阻塞合入）：
测试本次为无人值守运行，未构建或执行任何 PR 代码；以下证据来自对 PR 自身 CI 的 API 读取（按所审查的 commit）。 （CI 表格见上方标记区域，由 finalize 工作流在 CI 结束后原地更新。） 关于这套 CI 能证明什么、不能证明什么，有两点需要说明：
未验证：污染宿主的 before/after。PR 正文中 42 失败 → 52 通过、152 全过的表格是作者的本地运行结果，此处仅作为作者声明引用，未独立复跑。其与失败 run 的特征完全吻合（相同套件、相同数量、两条 Linux 腿）。 沙箱验证可以补齐剩余缺口： — Qwen Code · qwen3.8-max Reviewed at |
Collaborator
|
Confidence: 4/5 — minimal, well-reasoned fix for an observed main-CI failure; the open items are questions, not blockers. Stepping back: this is the autofix for #10330 landing exactly the repair the failure analysis calls for. The root cause is mechanical — an extra recall request shifting scripted request indices — and the fix removes the trigger at the harness level rather than patching individual suites, reusing the documented The reservations, such as they are: the Approval is deferred until CI lands green on this commit. 中文说明Confidence: 4/5——针对一次已观测到的 main CI 故障的最小且论证充分的修复；遗留项是问题，不是阻塞。 退一步看整体：这是 #10330 的 autofix PR，落地的正是故障分析所指向的修复。根因是机械性的——额外的 recall 请求使脚本化的请求序号整体错位——修复在框架层面移除了触发条件，而不是逐个修补套件，并且复用了文档化的 如有保留意见，则是：上文审查中提出的 批准将推迟到该 commit 的 CI 全绿之后执行。 — Qwen Code · qwen3.8-max Reviewed at |
Removing the run's scratch qwen dir threw when a CLI child that outlived its test was still writing under the debug directory: the removal walk reached a directory that refilled between its listing and its rmdir, and the ENOTEMPTY escaped teardown. Every test had passed, so the shard reported no failure and still exited red - the same shape as the memory-file restore before it was made best-effort. Retry the removal so an ordinary race resolves itself, and warn instead of throwing when it still cannot finish, so a host that keeps a directory busy stays diagnosable without failing the run. Sweep scratch homes an earlier run left behind, past an age floor that clears any run in flight on the same host, so the persistent pool does not accumulate what a best-effort cleanup gives up on. The new case drives the failure through a real writer holding the directory busy rather than a permission trick, so it does not depend on the runner's privileges, and it waits for that writer to be producing before tearing down - without the wait it passes against the very bug it pins.
… fix/e2e-hermetic-qwen-home
yiliang114
approved these changes
Aug 28, 2026
qqqys
approved these changes
Aug 28, 2026
Contributor
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
This was referenced Aug 28, 2026
Collaborator
|
Released in v0.22.3. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Gives every integration-test run its own global qwen dir instead of letting the spawned CLI read the one belonging to whoever is running the tests. The host's configuration is carried across so ambient auth keeps working, but none of the files that dir accumulates over time come with it — saved memories, tool-usage history, extensions, skills, and commands all start empty. A run that pins the global dir itself keeps its own and is left alone.
Why it's needed
The integration suites spawn the real CLI, and the CLI resolves the global qwen dir from the host. Hosted runners have an empty one, so nobody noticed; the persistent pool does not, and the saved memories left there by earlier jobs made managed auto-memory recall issue its own model request ahead of the agent's first turn.
The SDK suites script their fake model server by request index. That extra request shifted every index by one: each scripted tool call was answered to the recall selector instead of the turn under test, which then received only the trailing text. The visible symptom is a tool that never runs — no recorded tool call and a permission callback that is never invoked — across 42 cases in the permission-control and tool-control suites, on both Linux legs, while macOS and the other shards stayed green. The same failures reproduce on any developer machine that has saved memories, which is why this is worth fixing at the harness rather than per suite.
Configuration is carried over whole rather than filtered. The suites that talk to a real model rely on ambient auth, and that can live in the credentials block, in the environment block, or in the model routing — there is no subset that is safe to take alone, and the persistent pool's own credential source is not knowable from here. Narrowing the change to the accumulated files fixes the reds without gambling on where CI's auth comes from.
Reviewer Test Plan
How to verify
The failure depends on host state, so reproduce it by giving the run a home that looks like the persistent pool's: a scratch directory containing only a populated memories directory under the global qwen dir, with the tool caches symlinked back so the toolchain still works, then point the run at it. Do not pin the global qwen dir directly — that is the documented opt-out and it bypasses the fix.
Against that host state, confirm the permission-control and tool-control suites fail before the change and pass after, and that the deterministic no-key merge gate goes from red to fully green. Then confirm nothing regressed for a developer whose credentials live in the global settings: on your real home, the suite that talks to a real model should still authenticate and pass exactly as before, and the harness's own save/restore unit test should still pass.
Evidence (Before & After)
Not user-visible; test-harness only.
Both suites, against a host state shaped like the persistent pool's:
test:integration:no-ak:sandbox:none(merge gate)The before column matches the failing run exactly, which reported
Failed Tests 42in the same two suites on both Linux legs.No regression: the real-model single-turn suite still passes 15/15 on a real home with credentials in the global settings, and the harness's save/restore unit test still passes.
Tested on
Environment (optional)
Local runs of the integration suites without a sandbox, driving the bundled CLI against the repo's fake model server.
Risk & Scope
Linked Issues
Fixes #10330
中文说明
这个 PR 做了什么
让每次集成测试运行拥有自己的全局 qwen 目录，而不是让被拉起的 CLI 去读运行者本人的那一个。宿主的配置会被带过去，因此 ambient auth 照常可用；但这个目录长期累积下来的文件一个都不带——保存的记忆、工具使用历史、扩展、技能、命令全部从空开始。如果调用方自己指定了全局 qwen 目录，则保留其原样，不做干预。
为什么需要它
集成测试会拉起真实 CLI，而 CLI 的全局 qwen 目录来自宿主。Hosted runner 上这个目录是空的，所以一直没人察觉；持久化 runner 池上不是，那里由早先任务留下的记忆使得 managed auto-memory recall 在 agent 首轮之前先发出了一次自己的模型请求。
SDK 套件是按请求序号来编排其伪模型服务的。这次额外请求把每一个序号整体挪后一位：脚本里的每一次工具调用都被应答给了 recall selector，而真正被测的回合只拿到了兜底文本。可见症状是工具从未执行——没有记录到工具调用，权限回调也从未被触发——波及 permission-control 与 tool-control 两个套件的 42 个用例，两条 Linux 腿均如此，而 macOS 与其余分片保持全绿。同样的失败在任何存过记忆的开发机上都能复现，因此值得在测试基架层面修，而不是逐个套件去补。
配置是整份带过去的，而非做筛选。需要访问真实模型的套件依赖 ambient auth，而它可能位于凭据块、环境变量块或模型路由中——没有哪个子集是可以单独安全取用的，并且持久化 runner 池自身的凭据来源在此处也无法确证。把改动收窄到「累积的文件」这一面，既修好了这些红，又不必对 CI 的 auth 来源下赌注。
评审者测试计划
如何验证
这个失败依赖宿主状态，所以复现方式是给运行一个形似持久化 runner 池的 home：一个临时目录，其全局 qwen 目录下只放一份有内容的记忆目录，并把工具链缓存 symlink 回去以免破坏 toolchain，然后让运行指向它。不要直接指定全局 qwen 目录——那是本 PR 明确保留的退出通道，会绕过修复本身。
在该宿主状态下，确认 permission-control 与 tool-control 两个套件在改动前失败、改动后通过，并确认无密钥的确定性合并门从红变为全绿。随后确认对凭据存放在全局设置中的开发者没有回归：在你自己的真实 home 上，访问真实模型的套件应当仍能鉴权并通过，测试基架自身的保存/还原单元测试也应当仍然通过。
证据（改动前后）
非用户可见，仅涉及测试基架。
两个套件，在形似持久化 runner 池的宿主状态下：
test:integration:no-ak:sandbox:none（合并门）「改动前」一列与失败的那次运行完全一致：该运行在两条 Linux 腿上的同样两个套件中报告了
Failed Tests 42。无回归：在凭据位于全局设置的真实 home 上，真实模型的 single-turn 套件仍然 15/15 通过，测试基架的保存/还原单元测试也仍然通过。
测试环境
环境（可选）
在本地以无沙箱方式运行集成套件，驱动打包后的 CLI 对接仓库自带的伪模型服务。
风险与范围
关联 Issue
Fixes #10330