Skip to content

test(integration): wait for ACP command update notification - #10516

Merged
yiliang114 merged 1 commit into
mainfrom
codex/fix-acp-commands-update-wait
Aug 29, 2026
Merged

test(integration): wait for ACP command update notification#10516
yiliang114 merged 1 commit into
mainfrom
codex/fix-acp-commands-update-wait

Conversation

@yiliang114

@yiliang114 yiliang114 commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

What this PR does

This PR replaces the fixed one-second sleep in the ACP command-update integration check with condition-based polling. The check now stops as soon as available_commands_update arrives and keeps the existing assertions that validate the command payload.

Why it's needed

Main E2E run 33259149948 failed this check on all three attempts. The first two attempts reported ACP event-loop stalls of 1415.58 ms and 1190.13 ms, both longer than the fixed one-second wait, so the assertion inspected the notification list before the asynchronously queued update was delivered.

The failed run was triggered by the main commit that merged #9045, but #9045 did not change this ACP path. The fixed wait originated in #1113 and has also produced the earlier failure tracked by #10491. Runner load exposed the existing test race; this is not evidence that #9045 introduced an ACP product regression.

Reviewer Test Plan

How to verify

Run the focused ACP integration check with an OpenAI-shaped local test configuration. Confirm that session creation is followed by an available_commands_update, that the check returns as soon as the notification arrives, and that the init command assertions still pass. Repeating the focused check should not fail when delivery takes longer than one second but remains within the five-second bound.

Evidence (Before & After)

Before: the Linux sandbox:none shard failed three times with AssertionError: expected undefined to be defined at the command-update assertion. Two attempts logged event-loop stalls longer than the fixed wait.

After: npm run build and npm run bundle passed. The focused ACP check passed four consecutive local runs. git diff --check and the Prettier check also passed.

UI verification: N/A — this PR changes only integration-test synchronization and does not affect user-visible behavior.

Tested on

OS Status
🍏 macOS
🪟 Windows ⚠️
🐧 Linux ⚠️

Environment (optional)

Node.js 22.22.0, QWEN_SANDBOX=false, and a non-routable local OpenAI-shaped endpoint for the authentication-only setup. The full Linux CLI integration suite runs only for merge_group; the no-AK gate on the PR does not include this ACP file, so the focused local run is the direct verification until merge-queue execution.

Risk & Scope

  • Main risk or tradeoff: a real missing notification can take up to five seconds to fail instead of one second; successful delivery returns immediately.
  • Not validated / out of scope: the full E2E matrix was not run locally because it depends on repository secrets. No production ACP behavior is changed.
  • Breaking changes / migration notes: none.

Linked Issues

Fixes #10491
Fixes #10515

中文说明

本 PR 的改动

本 PR 将 ACP 命令更新集成检查中固定的一秒等待改为基于条件的轮询。收到 available_commands_update 后检查会立即继续,同时保留现有断言来验证命令数据。

为什么需要这个改动

main E2E run 33259149948 的该检查连续三次失败。前两次分别记录了 1415.58 毫秒和 1190.13 毫秒的 ACP 事件循环停顿,都超过了固定的一秒等待,因此异步排队的更新尚未送达时,断言就已经检查了通知列表。

失败 run 由合入 #9045 后的 main 提交触发,但 #9045 没有修改这条 ACP 路径。固定等待最早来自 #1113,之前 #10491 追踪的失败也属于同一问题。共享 runner 的负载暴露了既有的测试竞态,并不说明 #9045 引入了 ACP 产品回归。

Reviewer 测试计划

验证方式

使用 OpenAI 形态的本地测试配置运行聚焦的 ACP 集成检查。确认创建 session 后能够收到 available_commands_update,通知到达后检查立即继续,并且针对 init 命令的现有断言仍然通过。重复运行时,即使通知耗时超过一秒,只要在五秒上限内到达,检查也不应失败。

前后证据

改动前:Linux sandbox:none 分片在命令更新断言处连续三次以 AssertionError: expected undefined to be defined 失败,其中两次事件循环停顿超过固定等待时间。

改动后:npm run buildnpm run bundle 通过;聚焦的 ACP 检查在本地连续四次通过;git diff --check 和 Prettier 检查也通过。

UI 验证:不适用——本 PR 只修改集成测试同步方式,不影响用户可见行为。

测试平台

OS 状态
🍏 macOS
🪟 Windows ⚠️
🐧 Linux ⚠️

环境(可选)

Node.js 22.22.0、QWEN_SANDBOX=false,认证初始化使用不可路由的本地 OpenAI 形态端点。完整的 Linux CLI 集成测试仅在 merge_group 触发;PR 的 no-AK gate 不包含这个 ACP 文件,因此在 Merge Queue 执行前,聚焦的本地测试是这条路径的直接验证。

风险与范围

  • 主要风险或取舍:如果通知确实缺失,失败时间会由最多一秒变成最多五秒;正常收到通知时会立即继续。
  • 未验证或范围外:完整 E2E 矩阵依赖仓库 secrets,因此未在本地运行。本 PR 不修改生产 ACP 行为。
  • 破坏性变更或迁移说明:无。

关联 Issue

Fixes #10491
Fixes #10515

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
@yiliang114

yiliang114 commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator Author

E2E verification

  • npm run build: passed
  • npm run bundle: passed
  • Focused ACP command-update check: passed 4 consecutive runs on macOS with Node.js 22.22.0, QWEN_SANDBOX=false, and a non-routable local OpenAI-shaped endpoint
  • Sandboxed Linux verification on PR head 357ffbb: all 12 ACP integration tests passed
  • git diff --check and Prettier: passed
  • The full CLI suite is merge_group-only, and the no-AK gate on the PR does not include this ACP file
  • Post-merge E2E run 33267420074 was cancelled by a newer main push before it could provide a full-matrix result

The merged change preserves the notification payload and init command assertions; it only replaces the fixed sleep with bounded condition polling.

@qwen-code-ci-bot

qwen-code-ci-bot commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

Qwen Triage finishedview run. See the stage comments in this thread for the result.

Qwen Triage 已完成 —— 查看运行。结果见本线程中的各阶段评论。

@qwen-code-ci-bot

qwen-code-ci-bot commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the PR!

Template looks good ✓

Problem: observed, not theoretical. Main E2E run 33259149948 failed this check on all three attempts with AssertionError: expected undefined to be defined, and two attempts logged ACP event-loop stalls of 1415.58 ms and 1190.13 ms — both longer than the fixed one-second wait the check relied on. Tracked in #10491 and #10515.

Direction: aligned. This is the fixed-wait race #10491 already tracks; the wait originated in #1113 and #9045 never touched this path, so the framing — runner load exposed an existing test race, not a product regression — checks out. Fixing CI flakiness at the synchronization point is the right direction.

Size: not applicable — one integration-test file (+9/-2), no core paths touched.

Approach: minimal and right-sized — the diff replaces exactly one fixed sleep with bounded condition polling and keeps every assertion. Nothing to cut.

Risk: no elevated risk signals — test-only file, excluded from the high-risk path check.

Moving on to code review. 🔍

中文说明

感谢贡献!

模板完整 ✓

问题:已观测到,不是理论问题。main E2E run 33259149948 中该检查连续三次失败(AssertionError: expected undefined to be defined),其中两次记录了 1415.58 毫秒和 1190.13 毫秒的 ACP 事件循环停顿,均超过检查所依赖的固定一秒等待。已有 #10491#10515 追踪。

方向:对齐。这正是 #10491 追踪的固定等待竞态;该等待源自 #1113#9045 未触及这条路径,因此"共享 runner 负载暴露既有测试竞态,而非产品回归"的判断成立。在同步点修复 CI 抖动,方向正确。

规模:不适用——仅一个集成测试文件(+9/-2),未触及核心路径。

方案:最小且范围合理——diff 只把一处固定等待替换为有条件的有界轮询,所有断言保持不变。没有可砍的部分。

风险:无升级风险信号——纯测试文件,不在高风险路径检查范围内。

进入代码审查 🔍

Qwen Code · qwen3.8-max

Reviewed at 357ffbb0c56ca26a786a183b75670e0a1f5326c0 · re-run with @qwen-code /triage

@qwen-code-ci-bot

qwen-code-ci-bot commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

Code review

No blockers — re-verified against the base tree on this re-run, not carried over:

  • rig.poll(predicate, timeout, interval) in integration-tests/test-helper.ts matches the call shape used here, and the polling predicate is the exact same condition the following sessionUpdates.find(...) assertion checks — the two cannot diverge.
  • poll returns false on timeout rather than throwing, so the failure mode is unchanged: a genuinely missing notification still fails expect(commandsUpdate).toBeDefined(), now with a 5 s budget instead of 1 s — the tradeoff the PR description states.
  • The delay import is still used by three other waits in this file, so nothing dangles. The 5 s bound sits well inside the 60 s request timeout and the suite's 5-minute test timeout.
  • This is the established pattern — the exit-plan-mode test lower in the same file already polls the same way, including discarding the boolean result. That discard is the one non-blocking nit the automated review pass raised inline: on a timeout the failure surfaces as the generic assertion with no hint the bounded wait elapsed. True — but it matches the file's existing pattern, and it's optional to address.

One CI fact worth knowing (not a defect of this PR): the required no-AK integration gate runs a fixed file list that does not include cli/acp-integration.test.ts, and the Integration Tests (CLI, No Sandbox) shard that would run it is merge_group-only — skipped on every PR. So this PR's own CI cannot exercise the changed test; the proof lands post-merge on the main E2E suite, where the original failures were observed.

Test evidence — the PR's own CI via API (no PR code executed here)

CI on 357ffbb has settled. Three checks are not green, and none are attributable to this diff:

  • Test (ubuntu-latest, Node 22.x)cancelled by its own 60-minute job timeout (ran 16:28:18 → 17:28:58 UTC; the job declares timeout-minutes: 60). The job log ends with the runner terminating still-running npm run test:ci / node (vitest) / esbuild processes — the unit suite was still in flight when the timeout hit. This PR cannot affect that job: test:ci runs per-workspace unit tests only and never collects integration-tests/, where this one-file diff lives. The runner pool was visibly overloaded in these hours — a large share of sibling PR CI runs from the same window also landed cancelled.
  • Post Coverage Commentfailure, derivative: it posts coverage from an artifact the cancelled Test job never uploaded.
  • web-shell E2E Smokecancelled, collateral: the job declares needs: test.

Everything inside the PR's actual blast radius is green, including the no-AK integration gate.

Check Conclusion
Post Coverage Comment (ubuntu-latest, 22.x) ❌ failure
Test (ubuntu-latest, Node 22.x) 🚫 cancelled
web-shell E2E Smoke (ubuntu-latest, Node 22.x) 🚫 cancelled
Classify PR ✅ success
Dependency CVE audit ✅ success
Desktop Shell (ubuntu-22.04) ✅ success
Desktop Shell (windows-2022) ✅ success
Integration Tests (no-AK, No Sandbox) ✅ success
Secret scan (TruffleHog) ✅ success

One row per check name (latest run); skipped checks omitted; failures sort first. / 每个检查名一行(取最新一次运行),省略 skipped,失败项排在最前。

Sandboxed verification would settle the one claim CI cannot: @qwen-code /verify — a targeted run of the focused ACP command-update check, ideally on a Linux runner where the stalls were measured. As things stand, "the race is gone" rests on the author's four consecutive macOS passes (author's claim, not independently re-run) and the stall readings, because no PR-level check executes the changed file.

Real-scenario testing: N/A on the CI path — the change is integration-test synchronization only, nothing user-visible.

中文说明

代码审查

无阻塞问题——本次复跑重新对照基线代码核实,不是照搬上次结论:

  • integration-tests/test-helper.ts 中的 rig.poll(predicate, timeout, interval) 与此处调用形态一致;轮询谓词与后面 sessionUpdates.find(...) 断言的条件完全相同,二者不会分叉。
  • poll 超时返回 false 而不抛异常,失败模式不变:通知真正缺失时 expect(commandsUpdate).toBeDefined() 依然失败,只是预算从 1 秒变为 5 秒——正是 PR 描述中的取舍。
  • delay 导入仍被文件中另外三处等待使用,没有悬空导入。5 秒上限远小于 60 秒请求超时与套件 5 分钟的测试超时。
  • 这是既有模式——同文件下方的 exit-plan-mode 测试已采用同样的轮询,同样丢弃布尔返回值。自动审查流程在行内提出的唯一非阻塞建议正是针对这一点:超时时失败会表现为泛化断言,看不出有界等待已经耗尽。说法成立——但与文件既有模式一致,可改可不改。

一个值得了解的 CI 事实(不是本 PR 的缺陷):必需的 no-AK 集成门禁运行的是固定文件列表,其中不包含 cli/acp-integration.test.ts;能运行它的 Integration Tests (CLI, No Sandbox) 分片仅限 merge_group——在每个 PR 上都显示 skipped。因此本 PR 自身的 CI 无法执行被修改的测试;验证要在合并后的 main E2E 套件中落地,原始失败也正是发生在那里。

测试证据——通过 API 读取 PR 自身检查(此处未执行任何 PR 代码)

357ffbb 的 CI 已落定。三项检查未绿,均与本 diff 无关:

  • Test (ubuntu-latest, Node 22.x)——被自身 60 分钟的 job 超时取消(16:28:18 → 17:28:58 UTC,job 声明 timeout-minutes: 60)。job 日志结尾是运行器终止仍在运行的 npm run test:ci / node (vitest) / esbuild 进程——超时时单元测试仍在跑。本 PR 不可能影响该 job:test:ci 只运行各 workspace 的单元测试,从不收集本单文件 diff 所在的 integration-tests/。这些时段 runner 池明显过载——同一时间窗内大量兄弟 PR 的 CI 同样以取消告终。
  • Post Coverage Comment——失败,属衍生:它依赖被取消的 Test job 上传覆盖率产物,产物从未生成。
  • web-shell E2E Smoke——取消,受牵连:该 job 声明 needs: test

PR 实际影响范围内的检查全部为绿,包括 no-AK 集成门禁。

CI 结果表见上方英文部分(与上次相同,检查结论已落定)。

沙箱验证可以补齐 CI 无法覆盖的那一点:@qwen-code /verify——对聚焦的 ACP 命令更新检查做一次定向运行,最好在测得停顿的 Linux runner 上。目前"竞态已消除"由作者的四次 macOS 本地连续通过(作者陈述,未独立复现)和停顿读数支撑,因为 PR 级别的检查都不会执行被修改的文件。

真实场景测试:CI 路径不适用——本改动仅涉及集成测试同步,无用户可见行为变化。

Qwen Code · qwen3.8-max

Reviewed at 357ffbb0c56ca26a786a183b75670e0a1f5326c0 · re-run with @qwen-code /triage

@qwen-code-ci-bot

qwen-code-ci-bot commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

Confidence: 4/5 — a minimal, structurally correct fix for a well-evidenced race; the one reservation stays evidential, not technical — no PR-level check runs the changed test, so the final proof lands post-merge.

Stepping back: this is the kind of PR the gate should wave through. The problem is observed and measured — three consecutive failures, two stall readings above the one-second wait — and the root-cause story is internally consistent: the wait came from #1113, #9045 never touched this path. The fix is the smallest possible change: one sleep replaced by bounded polling on the exact condition the assertion already checks, reusing the rig's existing helper, assertions untouched. My independent proposal matches the diff; there is no simpler version of this fix, and in six months nobody will curse the author for it.

What this re-run adds: CI has settled, and the one thing to judge was the red column. Judged — Test hit its 60-minute job timeout mid-run on an overloaded pool (the job log shows vitest and esbuild still executing when the timeout reaped them), and that suite cannot even collect this PR's only changed file — test:ci never touches integration-tests/. The coverage-post failure and the smoke cancellation are its derivatives. Every check that can actually see this diff is green, and no PR runs remain pending. The deferred approval from the last pass had exactly this shape as its unblocking condition, so it is executed now — pinned to the reviewed commit.

Maintainer @wenshao already approved this exact commit; main requires two approving reviews, and this run records the second.

中文说明

置信度:4/5——对一个证据充分的竞态,这是最小且结构上正确的修复;唯一的保留在证据层面而非技术层面——PR 级别的检查都不会运行被修改的测试,最终验证要到合并后才落地。

退一步看:这正是门禁应当放行的 PR。问题是观测到且有度量的——连续三次失败,两次停顿读数超过一秒等待——根因叙述也自洽:等待源自 #1113#9045 未触及这条路径。修复是可能的最小改动:把一处固定等待替换为对断言已在检查条件本身的有界轮询,复用测试框架已有的助手函数,断言原封不动。我的独立方案与 diff 一致;这个修复不存在更简单的形态,半年后也不会有人抱怨它。

本次复跑的增量:CI 已落定,需要判断的就是那一列红色。判断结论——Test 是在运行器过载的时段被自身 60 分钟 job 超时在半途取消的(job 日志显示超时时 vitest 与 esbuild 仍在运行),而且该套件根本收集不到本 PR 唯一改动的文件——test:ci 从不触及 integration-tests/。覆盖率评论失败与冒烟测试取消都是它的衍生物。所有真正能看到本 diff 的检查均为绿色,PR 的 CI 运行也没有剩余未完成项。上一轮推迟批准所等待的正是这种解锁形态,因此现在执行批准——并钉在被审提交上。

维护者 @wenshao 已批准同一提交;main 需要两个批准,本次运行记录第二个。

Qwen Code · qwen3.8-max

Reviewed at 357ffbb0c56ca26a786a183b75670e0a1f5326c0 · re-run with @qwen-code /triage

@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

No blocking findings. The change correctly replaces the fixed 1-second delay with condition-based polling, preserves the existing command-payload assertions, and remains isolated to the single ACP test session. The reported build, bundle, formatting, and focused integration checks also pass.

@yiliang114

Copy link
Copy Markdown
Collaborator Author

@qwen-code-review-bot approve this pr if no blockers

@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

I’m not approving yet because the required CI checks are still running. No code blockers are currently evident; approval can proceed once the checks finish successfully.

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially reviewed — gaps disclosed. Suggestions are inline.

Not reviewed: reverse audit — stopped before round 1 by the review time budget.

中文说明

仅完成部分审查,审查缺口已披露。 建议见行内评论。

未审查:反向审计——评审时间预算不足,未能开始第 1 轮。

— qwen3.8-max via Qwen Code /review (v0.22.3)

Comment thread integration-tests/cli/acp-integration.test.ts
Comment thread integration-tests/cli/acp-integration.test.ts
@wenshao

wenshao commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

@qwen-code /triage

@qwen-code-ci-bot

qwen-code-ci-bot commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

Sandboxed verification: ✅ passed — merge-ready (agent verdict) - workflow run

Ran the PR in an isolated, token-free container: A/B against the base build, mock-free harness assertions, targeted gates. Advisory evidence for human reviewers — not a review, an approval, or a CI check.

Scripted assertions: 200 passed · 0 failed · 200 total

Flakiness gate: not applicable — no runnable changed test files (1 out-of-scope file(s) noted in the log)

中文 — 判定:✅ 通过 · 可合入(agent 判定)

沙箱验证在隔离、无凭证的容器中执行了该 PR 的代码(与 base 构建 A/B 对照、无 mock harness 断言、定向门禁)。仅作为评审证据,不构成评审、批准或 CI 检查

脚本断言:200 通过 · 0 失败 · 200 总计

抖动门:不适用 — no runnable changed test files (1 out-of-scope file(s) noted in the log)

Verification report

PR #10516 — test(integration): wait for ACP command update notification

Verdict: merge-ready — 200/200 scripted assertions passed, 0 unexpected failures. Verified head: 357ffbb0c56ca26a786a183b75670e0a1f5326c0 (merge-ref checkout HEAD^2; single commit, matches the snapshot's commits array).

中文摘要
  • 结论merge-ready。200/200 条脚本化断言通过,0 条意外失败。
  • A/B 结论:中心主张成立且被证明 load-bearing。用 SIGSTOP/SIGCONT 物理停顿真实 ACP agent 的事件循环(与 PR 引用的 1415.58 ms 停顿同形态),在送达延迟落入 (1s, 5s] 的 8 个翻转单元中,base 的 delay(1000) 全部以"update 缺失"形状变红,head 的 poll(5000, 100) 全部变绿且 waitMs ≈ dActual + 100ms(通知一到即返回,见下表与 01-ab-sigstop-base-vs-head.png)。其中一次运行里两个无注入的 natural 单元也复现了 CI 失败形态(送达 1696 ms / 1108 ms,agent stderr 记录 [perf] acp agent event loop stall: max=1454.37ms,与 PR 引用的仪表一致)。通知真正缺失时 head 仍在 ~5.1 s 以同一断言形状失败(变异对照,02-mutation-emission-killed.png),轮询不会掩盖真实回归。
  • Findings:1 条非阻塞完整性观察——同文件另两处固定等待(acp-integration.test.ts:1007delay(2000):1102delay(500))属同一竞态类别,本 PR 未覆盖。
  • 未覆盖:完整 E2E 矩阵 / Windows / macOS(与 PR 相同,需 secrets);兄弟固定等待未做停顿测量(仅观察);eslint 只跑了干净检查未做活体探针(prettier 与 tsc 均做了)。

Central claim + A/B

Central claim: replacing the fixed await delay(1000) with rig.poll(pred, 5000, 100) makes the available_commands_update check tolerate delivery delays > 1 s (returning as soon as the notification arrives) while still failing — within ~5 s — when the notification is genuinely absent.

How the A/B drives the real code: ab-sigstop.mjs (in this dir) speaks stdio JSON-RPC to the real bundled CLI (dist/cli.js --acp, built at HEAD via npm run bundle) exactly like setupAcpTest does, lifts both wait strategies verbatim (base delay(1000); head TestRig.poll from test-helper.ts:586-624), and physically stalls the agent's event loop with SIGSTOP/SIGCONT immediately after the session/new response — the same failure shape the PR cites (agent-side stalls of 1415.58 ms / 1190.13 ms). Every cell's expected verdict is derived from one universal law, not per-cell cherry-picking: base passes iff the measured delivery dActual ≤ 1000 ms; head passes iff dActual ≤ 5000 ms (±25 ms boundary band). Base cells keep listening after their frozen 1 s verdict so dActual is measured post-hoc. Flip cells carry a scripted validity control requiring dActual ∈ (1000, 5000], with auto-retry otherwise.

cell (run A) strategy stall dActual (ms) verdict note
natural-1 base none 1696 RED no injection; agent stderr [perf] … stall: max=1454.37ms — the CI failure shape, reproduced naturally on this loaded runner
natural-1 head none 1524 green poll adapted
natural-2 base / head none 567 / 1594 green / green
natural-3 base none 1108 RED second natural reproduction
natural-3 head none 496 green
stall-500 base / head 0.5 s 734 / 718 green / green positive control, both arms
stall-1500 base 1.5 s 1915 RED flip
stall-1500 head 1.5 s 4136 green waitMs=4222 ≈ dActual+86ms
stall-2500 base / head 2.5 s 3487 / 3463 RED / green flip
stall-4000 base / head 4 s 4507 / 4550 RED / green flip
absent base 7 s 7246 RED expected red
absent head 7 s 7662 RED at 5089 ms absence still detected, within the PR's stated ~5 s fail budget

Run A: 89/89 scripted assertions (ab-run2.log). A second full matrix run (captured as 01-ab-sigstop-base-vs-head.png) scored 95/95 and shows the same flips at dActual 1813/2681/4305 ms (base RED, head green) plus stall-500-head green at dActual 1212 ms — poll absorbing a >1 s delivery that the base law would fail. Across both runs: 8 flip cells (delivery in (1 s, 5 s]) with base red / head green, 0 exceptions to the universal law over 36 strategy cells.

Witnesses: evidence/01-ab-sigstop-base-vs-head.png (matrix as printed), evidence/02-mutation-emission-killed.png (mutation control).

Mutation control (vacuity): in a scratch copy of the bundle, the single emission site's literal was renamed (sessionUpdate: "available_commands_update"…_MUT in dist/chunks/acpAgent-*.js; real dist/ verified untouched). Against it, both test versions fail via the missing-update assertion — base at 1001 ms, head at 5047–5092 ms (7/7 assertions, two runs). So: the scenario reaches the code under test, the poll hides nothing, the test change loses no coverage (mutant killed on both sides, no killed→survived regression), and the harness demonstrably can turn the head arm red. The killing assertion in both versions is the unchanged expect(commandsUpdate).toBeDefined() — the changed call site ignores poll()'s boolean return, and it is that subsequent expect that fails on absence; the mutation cells prove this live.

End-to-end suite A/B (real vitest file, QWEN_SANDBOX=false, same runner, same bundle — justified because git diff HEAD^1..HEAD touches only the test file, so the bundle is byte-identical by construction): head run 1 12/12 (229 s), head run 2 12/12 (112 s), base arm 12/12 (181 s). The base pass is consistent with the race being load-correlated: on run A's natural cells this runner delivered at 1696/1108 ms and base went red; during the quieter base-suite run delivery stayed <1 s. That is exactly the PR's premise — runner load exposes a pre-existing race; the polling wait removes the load dependence for deliveries ≤ 5 s.

Reviewer Test Plan walkthrough

plan step result
focused ACP check passes; session creation followed by available_commands_update ✓ head vitest 12/12 twice; harness natural cells 6/6 head-green
check returns as soon as the notification arrives returned-promptly assertions: waitMs ≈ dActual + ≤100 ms (e.g. 4633 vs 4550), never the full 5000 on pass
init command assertions still pass init-defined / init-description true in every found cell (payload assertions unchanged in the diff)
repeating the check should not fail when delivery is 1–5 s ✓ 8 flip cells head-green + two consecutive 12/12 suite runs

Findings

  1. Sibling fixed waits of the same race class remain in the file (completeness, non-blocking). integration-tests/cli/acp-integration.test.ts:1007 (await delay(2000) "give time for tool calls to be processed") and :1102 (await delay(500) before reading usage metadata) wait for asynchronously delivered session/update notifications with the same fixed-sleep pattern this PR removes from the command-update check. Not stall-measured this round (observation only) — under runner load they can observe the same late-delivery shape. The repo already contains the correct pattern (waitForSessionUpdate with deadline + throw in acp-cron.test.ts:287-303). The PR is explicitly scoped to the command-update check, so this is a follow-up candidate, not a merge condition.

Not covered

  • Full E2E matrix, Windows, macOS lanes — same exclusion as the PR (needs repo secrets).
  • The sibling waits above were not exercised under induced stalls; finding 1 is an observation, not a measured claim.
  • ESLint ran clean on the changed file but without a planted-violation live proof (prettier and tsc -p integration-tests both got live proofs: a misformatted probe and a TS2322 probe were each flagged, then removed, then clean).
  • Trial merge into a newer main: not needed — HEAD^1 equals the snapshot's baseRefOid (PR is current as of this round).
  • Deliveries > 5 s still fail at head (the PR's named, accepted tradeoff). Worst observed delivery under a 4 s injected stall was 4771 ms on the loaded runner; margin to the bound is real but finite.

Methodology

Environment: CI merge-ref checkout (HEAD merge commit, HEAD^1=base 6606362, HEAD^2=verified head 357ffbb), node:22-bookworm container, shared loaded runner; npm ci + npm run build pre-ran, npm run bundle run by this round (bundle-head.log). Harnesses (ab-sigstop.mjs) drive the compiled dist/cli.js over real stdio JSON-RPC with per-run isolated QWEN_HOME; stalls are physical (SIGSTOP/SIGCONT on the agent process) and corroborated by the agent's own [perf] acp agent event loop stall stderr instrumentation. The base-arm suite ran from a scratch git worktree of HEAD^1 with node_modules and dist symlinked to the head tree — valid because the diff contains zero production changes (verified with git diff HEAD^1..HEAD --stat -- packages/); worktree removed after capture. Mutation control used a scratch copy of dist/ (removed after); the real dist/ was verified untouched. Raw logs: ab-run2.log, mutation-run.log, vitest-head-1.log, vitest-head-2.log, vitest-base-1.log in this dir. Assertion tally: harness runs 89 + 95 + 7 + 7 = 198, plus 2 scripted suite assertions (head vitest runs, each "12/12 passed" parsed from the summary with exit 0) = 200; the base-arm suite run is a qualitative control cell and is deliberately excluded from the count. Expected reds (base flip cells, mutation cells, absence cells) are encoded as passing scripted assertions about the predicted verdict; fail counts only unexpected outcomes.

Flakiness gate log

integration test, out of gate scope: integration-tests/cli/acp-integration.test.ts

verdict: n/a
summary: no runnable changed test files (1 out-of-scope file(s) noted in the log)

Evidence images

01-ab-sigstop-base-vs-head

02-mutation-emission-killed

Harness scripts and raw logs are in the workflow run artifacts (7-day retention).

Qwen Code · sandboxed verification

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, looks ready to ship. ✅

@yiliang114
yiliang114 added this pull request to the merge queue Aug 29, 2026
Merged via the queue into main with commit 1fb4a4e Aug 29, 2026
109 of 112 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Main CI failed: E2E Tests on f8bcdaae8f78 Main CI failed: E2E Tests on 379bef96030e

4 participants