Skip to content

fix(test): give the serve routes daemon a CI-sized ACP handshake budget - #10846

Merged
wenshao merged 1 commit into
mainfrom
fix/e2e-serve-acp-init-timeout
Sep 2, 2026
Merged

fix(test): give the serve routes daemon a CI-sized ACP handshake budget#10846
wenshao merged 1 commit into
mainfrom
fix/e2e-serve-acp-init-timeout

Conversation

@yiliang114

Copy link
Copy Markdown
Collaborator

What this PR does

This PR raises the ACP initialize handshake budget for the daemon that integration-tests/cli/qwen-serve-routes.test.ts spawns, from the 10-second product default to 60 seconds, by passing --initialize-timeout-ms on the harness's qwen serve command line. No assertion in the file depends on that budget, and the product default is untouched.

Why it's needed

Run 33633418567 on 76c32fd5 turned main red on one shard: E2E Test (Linux) - sandbox:docker - shard 3/3, cli/qwen-serve-routes.test.ts > qwen serve — POST /session validation + concurrent coalescing > honors and reserves a normalized caller-supplied session ID, AssertionError: expected 504 to be 200 at line 685.

The 504 is not a route defect. The underlying error is DaemonHttpError: POST /session: AcpSessionBridge initialize timed out after 9993ms, and the test burned 36.5s across three attempts (retry x2) — three consecutive ~10s handshake timeouts against DEFAULT_INIT_TIMEOUT_MS = 10_000 in packages/acp-bridge/src/bridge.ts. The same commit's three sandbox:none shards all passed.

This is the second occurrence of that exact signature, and both landed on the sandbox:docker leg: #10605's investigation attributed run 33351032808 to "three 10-second ACP channel initialization timeouts" (it surfaced as HTTP 500 back then; #10404 has since reclassified initialization timeouts to 504). The daemon has to spawn a Node child that loads the CLI bundle and answer initialize within 10s, on a shard that shares a 128-core ECS host with roughly 30 other logical runners. Ten seconds is a budget for an interactive desktop; under that contention it is a coin flip, and vitest's two retries make it a triple coin flip rather than a fix.

The alternative — treating a slow subprocess handshake on an oversubscribed shared runner as a product bug — is not supported by the evidence: nothing in the failure points at the route, the session-ID normalization it exercises, or the daemon's behavior once the child answers.

Reviewer Test Plan

How to verify

  1. Read the diff: the spawned daemon gains --initialize-timeout-ms 60000. packages/cli/src/commands/serve.ts defines the flag as a number and run-qwen-serve.ts only rejects values above the maximum JS timer delay, so 60000 is accepted and validated. The value is not part of /capabilities, so the file's exact-equality capabilities baseline is unchanged.
  2. npm run build && npm run bundle, then npx vitest run --root ./integration-tests cli/qwen-serve-routes.test.ts — the suite passes with the daemon booting normally; the raised budget only changes how long a handshake may take before it is abandoned.
  3. Watch the sandbox:docker shard that owns this file over the next few main runs and expect no AcpSessionBridge initialize timed out failure.

Evidence (Before & After)

Before: expected 504 to be 200 at cli/qwen-serve-routes.test.ts:685, cause AcpSessionBridge initialize timed out after 9993ms, 36534ms across three attempts, shard duration 1061.93s, on ecs-qwen-hk5-30.

After: not reproducible locally — the failure needs the shared ECS host's contention. The change is confined to the harness's command line and is covered by the run above.

Tested on

OS Status
🍏 macOS N/A
🪟 Windows N/A
🐧 Linux ⚠️

Environment (optional)

Prettier and source inspection only; the E2E leg that exhibits the failure runs post-merge on main.

Risk & Scope

  • Main risk or tradeoff: --initialize-timeout-ms raises every ACP child request timeout in that daemon, not just the handshake, so a genuinely hung child in this file would now take 60s to surface instead of 10s. No test here asserts a timeout, and the file's slowest case is bounded by vitest's own timeouts.
  • Not validated / out of scope: the other integration tests that spawn qwen serve (_daemon-harness.ts and its callers) keep the 10s default — they have not shown this signature, and widening the change without evidence would be speculative. If the same failure appears there, the harness is the place to fix it once.
  • Breaking changes / migration notes: none. The product default stays 10s; only this test's daemon is configured.

Linked Issues

Fixes #10833

中文说明

本 PR 做了什么

本 PR 把 integration-tests/cli/qwen-serve-routes.test.ts 启动的 daemon 的 ACP initialize 握手预算,从产品默认的 10 秒提高到 60 秒,方式是在 harness 的 qwen serve 命令行上传入 --initialize-timeout-ms。该文件中没有任何断言依赖这个预算,产品默认值也未改动。

为什么需要

76c32fd5 上的 run 33633418567 让 main 变红,失败的是一个 shard:E2E Test (Linux) - sandbox:docker - shard 3/3,用例 cli/qwen-serve-routes.test.ts > qwen serve — POST /session validation + concurrent coalescing > honors and reserves a normalized caller-supplied session ID,报错 AssertionError: expected 504 to be 200(第 685 行)。

504 并不是路由缺陷。底层错误是 DaemonHttpError: POST /session: AcpSessionBridge initialize timed out after 9993ms,该用例在三次尝试(retry x2)中共耗时 36.5 秒 —— 即针对 packages/acp-bridge/src/bridge.tsDEFAULT_INIT_TIMEOUT_MS = 10_000 的三次连续约 10 秒握手超时。同一 commit 的三个 sandbox:none shard 全部通过。

这是该特征第二次出现,且两次都落在 sandbox:docker 通道上:#10605 的排查曾把 run 33351032808 归因为"三次 10 秒的 ACP channel 初始化超时"(当时表现为 HTTP 500,#10404 之后把初始化超时重新分类为 504)。daemon 需要派生一个加载 CLI bundle 的 Node 子进程,并在 10 秒内回应 initialize,而这个 shard 与大约 30 个其它逻辑 runner 共享一台 128 核 ECS 宿主。10 秒是给交互式桌面场景的预算;在这种争抢下它就是抛硬币,而 vitest 的两次重试只是把它变成抛三次硬币,并不是修复。

另一种解释 —— 把超载共享 runner 上子进程握手变慢当作产品缺陷 —— 没有证据支持:失败信息里没有任何一处指向该路由、它验证的 session ID 归一化,或子进程回应之后 daemon 的行为。

Reviewer Test Plan

如何验证

  1. 阅读 diff:被派生的 daemon 增加了 --initialize-timeout-ms 60000packages/cli/src/commands/serve.ts 将该 flag 定义为 number,run-qwen-serve.ts 只拒绝超过 JS 定时器最大延迟的值,因此 60000 会被接受并通过校验。该值不属于 /capabilities,所以该文件的 capabilities 精确相等断言不受影响。
  2. npm run build && npm run bundle,然后 npx vitest run --root ./integration-tests cli/qwen-serve-routes.test.ts —— daemon 正常启动、用例通过;提高的预算只改变握手在被放弃前允许的时长。
  3. 观察后续几次 main run 中承载该文件的 sandbox:docker shard,预期不再出现 AcpSessionBridge initialize timed out 失败。

Evidence(修复前后)

修复前:cli/qwen-serve-routes.test.ts:685expected 504 to be 200,原因为 AcpSessionBridge initialize timed out after 9993ms,三次尝试共 36534ms,shard 总时长 1061.93s,运行于 ecs-qwen-hk5-30

修复后:本地无法复现 —— 该失败依赖共享 ECS 宿主的争抢。改动仅限于 harness 的命令行,由上述 run 提供依据。

测试环境

OS Status
🍏 macOS N/A
🪟 Windows N/A
🐧 Linux ⚠️

Environment(可选)

仅做了 Prettier 与源码检查;出现该失败的 E2E 通道在合入 main 后才运行。

风险与范围

  • 主要风险或取舍:--initialize-timeout-ms 会提高该 daemon 中所有 ACP 子进程请求的超时,而不只是握手,因此该文件中若真的出现子进程挂死,暴露时间会从 10 秒变为 60 秒。这里没有任何用例断言超时行为,且该文件最慢的情形受 vitest 自身超时约束。
  • 未验证 / 不在范围内:其它派生 qwen serve 的集成测试(_daemon-harness.ts 及其调用方)保持 10 秒默认值 —— 它们没有出现过该特征,在没有证据的情况下扩大改动属于臆测。如果同样的失败出现在那里,harness 才是一次性修好的地方。
  • 破坏性变更 / 迁移说明:无。产品默认值仍为 10 秒,只有这个测试的 daemon 被显式配置。

关联 Issue

Fixes #10833

https://claude.ai/code/session_01MCE9CXnMVrX4fUxHpQoUr8

The ACP child's initialize handshake gets 10s by default, a budget sized
for an interactive desktop rather than for a shard sharing a 128-core ECS
host. The sandbox:docker leg loses a route test to three ~10s handshake
timeouts and a 504 while the same commit's sandbox:none shards pass.

Nothing in the file asserts the handshake budget, so raise it to 60s for
the daemon the harness spawns.

Claude-Session: https://claude.ai/code/session_01MCE9CXnMVrX4fUxHpQoUr8
@qwen-code-ci-bot

qwen-code-ci-bot commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

⚠️ Deferred approval withheld — 1 PR CI workflow run(s) on 2013b22 did not finish green; see the updated table in the Stage 2 comment. Re-run @qwen-code /triage after fixes. finalize run

⚠️ 延迟审批已搁置 —— 2013b22 有 1 个 PR CI workflow 未以绿色完成,详见 Stage 2 评论中已更新的表格。修复后可重新运行 @qwen-code /triage查看 finalize 运行

@qwen-code-ci-bot

Copy link
Copy Markdown
Collaborator

Thanks for the PR!

Template looks good ✓

Problem: observed failure with solid evidence — verified against the actual job log of run 33633418567: E2E Test (Linux) - sandbox:docker - shard 3/3 lost honors and reserves a normalized caller-supplied session ID to AcpSessionBridge initialize timed out after 9993msexpected 504 to be 200, 36534ms across three attempts. That is the second occurrence of this signature (first: run 33351032808, diagnosed in #10605), both on the sandbox:docker leg. Linked issue #10833 is open.

Direction: aligned. This keeps main green under an oversubscribed shared runner and does not touch product behavior — the product default stays 10s. CHANGELOG has no direct reference to CI harness budgets, but the daemon/ACP handshake reliability area is relevant.

Size: not applicable — no core paths touched; 11 added lines, all in one integration test file.

Approach: the scope feels right. Raising the budget only for the daemon this file spawns, through the existing --initialize-timeout-ms flag, is the minimal fix; raising the product default would mask genuine hangs, and more retries would be three coin flips instead of one. Leaving the other qwen serve harnesses on 10s until they show the same signature is the right call.

Risk: no elevated risk signals — the changed file is a test file, no high-risk-path match.

Moving on to code review. 🔍

中文说明

感谢贡献!

模板完整 ✓

问题:已观测到的失败,证据扎实 —— 已从 run 33633418567 的实际 job 日志核实:E2E Test (Linux) - sandbox:docker - shard 3/3honors and reserves a normalized caller-supplied session ID 用例以 AcpSessionBridge initialize timed out after 9993msexpected 504 to be 200 失败,三次尝试共 36534ms。这是该特征第二次出现(第一次为 run 33351032808,由 #10605 诊断),两次都落在 sandbox:docker 通道。关联 issue #10833 处于 open 状态。

方向:对齐。这是让 main 在高争抢的共享 runner 上恢复绿色,不涉及产品行为 —— 产品默认值保持 10 秒。CHANGELOG 没有对 CI harness 预算的直接引用,但 daemon/ACP 握手可靠性这个领域是相关的。

规模:不适用 —— 未触及核心路径;11 行新增全部在一个集成测试文件内。

方案:范围合理。通过现有的 --initialize-timeout-ms flag 只提高该文件所启动的 daemon 的预算,是最小修复;提高产品默认值会掩盖真正的挂起,增加重试次数只是抛三次硬币。让其它 qwen serve harness 保持 10 秒、直到出现同样特征再处理,是合理的取舍。

风险:无升级风险信号 —— 改动文件是测试文件,未命中高风险路径。

进入代码审查 🔍

Qwen Code · qwen3.8-max

Reviewed at 2013b22829a01d3efffd7a8dfaf88ae4ca164137 · re-run with @qwen-code /triage

@qwen-code-ci-bot

qwen-code-ci-bot commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Code review

Reviewed against the base branch. The flag wiring is real and the value is accepted end to end: serve.ts declares --initialize-timeout-ms as a number, run-qwen-serve.ts validates it as a positive integer below the JS timer ceiling, and the bridge consumes it as initTimeoutMs — 60_000 passes all three layers. The spawn-array insertion sits right after --workspace where it belongs, and the constant plus its comment carry the run numbers that justify the value.

No assertion in the file depends on the handshake budget — the only timeout literals there are the boot-readiness timer and a 10s extension-operation poll, both independent of the daemon's ACP budget. Also checked the bound the PR cites: the suite's vitest testTimeout is 5 minutes (integration-tests/vitest.config.ts), so a 60s handshake still fits with room to spare, and the config's retry: 2 matches the three-attempt shape of the observed failure.

One wording nit in the PR description, non-blocking: the value does flow into /capabilities — the bootstrap envelope advertises limits.sessionRestoreTimeoutMs, which resolves as max(initializeTimeoutMs, 60_000). The conclusion still holds for two reasons: at 60_000 the advertised value is unchanged (it equals the restore default), and the file's exact-equality baseline only asserts the features array (plus v/mode), never limits.

The disclosed tradeoff is real: initTimeoutMs is reused across the bridge as the budget for new-session creation, teardown, and other child requests, so a genuinely hung child in this suite would now surface after 60s instead of 10s. That is bounded by vitest's own timeout, and nothing here blocks.

Testing evidence — the PR's own CI, read via API (no PR code executed in this run)

CI on the reviewed commit so far: nothing red. The two suites that actually execute this file — Test (ubuntu-latest, Node 22.x) (unit) and Integration Tests (no-AK, No Sandbox) — are still running; the finalize pass updates the table below when they land. macOS/Windows legs are skipped by the workflow filter, not failed.

Final CI results for 2013b22 (auto-updated by the triage finalize job after CI completed):

Check Conclusion
Test (ubuntu-latest, Node 22.x) ❌ failure
web-shell E2E Smoke (ubuntu-latest, Node 22.x) 🚫 cancelled
Classify PR ✅ success
Dependency CVE audit ✅ success
Desktop Shell (ubuntu-22.04) ✅ success
Desktop Shell (windows-2022) ✅ success
Integration Tests (no-AK, No Sandbox) ✅ success
Post Coverage Comment (ubuntu-latest, 22.x) ✅ success
Secret scan (TruffleHog) ✅ success

One row per check name (latest run); skipped checks omitted; failures sort first. / 每个检查名一行(取最新一次运行),省略 skipped,失败项排在最前。

One honest caveat about the central claim: the flake only reproduces under the shared ECS host's contention on the post-merge sandbox:docker leg, so no pre-merge lane — this PR's CI included — can prove the flake is gone; the real oracle is watching that shard over the next main runs, exactly as the PR's own test plan commits. What the running CI does substantiate is the claim it can reach: the suite passes with the raised budget on its spawned daemon.

中文说明

代码审查

已对照基线分支核实。flag 接线真实且取值端到端被接受:serve.ts--initialize-timeout-ms 声明为 number,run-qwen-serve.ts 校验其为低于 JS 定时器上限的正整数,bridge 以 initTimeoutMs 消费 —— 60_000 通过全部三层。spawn 参数插入在 --workspace 之后的正确位置,常量及注释记录了支撑该取值的 run 编号。

文件中没有任何断言依赖握手预算 —— 仅有的超时字面量是启动就绪计时器和 10 秒的扩展操作轮询,都与 daemon 的 ACP 预算无关。也核对了 PR 所述的上界:该套件 vitest testTimeout 为 5 分钟(integration-tests/vitest.config.ts),60 秒握手仍有余量;配置里的 retry: 2 与观测到的三次尝试形态一致。

PR 描述中有一处措辞小问题,不阻塞:该值确实会流入 /capabilities —— bootstrap 信封会广播 limits.sessionRestoreTimeoutMs,其取值为 max(initializeTimeoutMs, 60_000)。但结论依然成立,原因有二:取 60_000 时广播值不变(恰等于 restore 默认值);且该文件的精确相等基线只断言 features 数组(及 v/mode),从不断言 limits

已披露的取舍是真实的:initTimeoutMs 在 bridge 中被复用为新建会话、teardown 及其它子进程请求的预算,因此本套件里真正挂死的子进程现在会在 60 秒后才显现而非 10 秒。该上界受 vitest 自身超时约束,无阻塞项。

测试证据 —— 该 PR 自己的 CI,通过 API 读取(本次运行未执行任何 PR 代码)

目前该提交上的 CI:无红色。真正执行本文件的两个套件 —— Test (ubuntu-latest, Node 22.x)(单测)与 Integration Tests (no-AK, No Sandbox) —— 仍在运行;完成后 finalize 流程会更新下表。macOS/Windows 通道是工作流过滤跳过的,不是失败。

关于核心结论的一点诚实说明:该 flake 只在共享 ECS 宿主争抢下的 post-merge sandbox:docker 通道复现,任何 pre-merge 通道(包括本 PR 的 CI)都无法证明 flake 已消失;真正的 oracle 是观察后续若干次 main run 中该 shard 的表现 —— 这也正是 PR 测试计划自己的承诺。当前 CI 能证实的是它够得着的结论:提高预算后该套件在其启动的 daemon 上通过。

Qwen Code · qwen3.8-max

Reviewed at 2013b22829a01d3efffd7a8dfaf88ae4ca164137 · re-run with @qwen-code /triage

@qwen-code-ci-bot

Copy link
Copy Markdown
Collaborator

Confidence: 4/5 — clean, evidence-backed deflake; the only nit is the PR description's imprecise sentence about /capabilities (named in the review comment).

Stepping back: the failure is real — verified from the actual job log, not just the PR's retelling — and the diagnosis matches what the code does: the bridge spawns a child that loads the CLI bundle and must answer initialize within initTimeoutMs, defaulting to 10s. The fix is exactly what I would have proposed independently: raise the budget through the existing flag, for the daemon this one suite spawns, product default untouched. No simpler path was missed — raising the product default would mask real hangs, and more retries are three coin flips instead of one. Eleven lines, all of them necessary, with a comment that records the exact runs to check against; in six months, if the signature recurs, that comment is exactly what I want to find there.

Why not 5: the description says the value "is not part of /capabilities", but it does flow into the envelope — as limits.sessionRestoreTimeoutMs, resolved as max(initializeTimeoutMs, 60_000). At the chosen 60_000 the conclusion (baseline unchanged) still holds, so this is a prose nit, not a defect in the diff.

The one thing nobody can know pre-merge is whether the flake is actually gone — that needs the shared ECS host's contention, which only the post-merge sandbox:docker leg sees. The real check is the next few main runs, as the test plan commits.

Approval is deferred until CI lands green on this commit; the finalize pass will pin it to the reviewed SHA.

中文说明

置信度:4/5 —— 干净、有证据支撑的去 flake;唯一的 nit 是 PR 描述中关于 /capabilities 的那句不精确表述(已在审查评论中点名)。

退后一步看:失败是真实的 —— 已从实际 job 日志核实,而非仅凭 PR 转述 —— 诊断与代码行为一致:bridge 派生加载 CLI bundle 的子进程,必须在 initTimeoutMs(默认 10 秒)内回应 initialize。修复正是我独立会提出的方案:通过现有 flag 提高预算,只作用于这一个套件启动的 daemon,产品默认值不动。没有更简的路径被遗漏 —— 提高产品默认值会掩盖真正的挂起,增加重试只是抛三次硬币。11 行改动全部必要,注释记录了可对照的 run 编号;六个月后若该特征再现,这正是我想找到的注释。

为什么不是 5:描述里说该值"不属于 /capabilities",但它确实流入信封 —— 以 max(initializeTimeoutMs, 60_000) 解析后作为 limits.sessionRestoreTimeoutMs。在所取 60_000 下结论(基线不变)依然成立,所以这是措辞 nit,不是 diff 缺陷。

pre-merge 无法知道的一件事是 flake 是否真的消失 —— 那需要共享 ECS 宿主的争抢,只有 post-merge 的 sandbox:docker 通道具备。真正的检验是后续几次 main run,如测试计划所承诺。

批准推迟到该提交的 CI 变绿;finalize 流程会把批准钉在所审查的 SHA 上。

Qwen Code · qwen3.8-max

Reviewed at 2013b22829a01d3efffd7a8dfaf88ae4ca164137 · re-run with @qwen-code /triage

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Downgraded from Approve to Comment: CI still running. Partially reviewed — gaps disclosed.

Not reviewed: reverse audit — stopped before round 1 by the review time budget.

Test Plan (not a blocker): cli/qwen-serve-routes.test.ts:685no such file or directory.

中文说明

⚠️ 已从批准降级为评论:CI still running。 仅完成部分审查,审查缺口已披露。

未审查:反向审计——评审时间预算不足,未能开始第 1 轮。

Test Plan(非阻断):cli/qwen-serve-routes.test.ts:685no such file or directory

— qwen3.8-max via Qwen Code /review (v0.22.3)

@chiga0 chiga0 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings.
Approval blockers: none.

Tier: Scan (test-only change, one file, 11 lines added — a constant with explanatory comment and one extra argument to the daemon spawn).

Checked:

  • --initialize-timeout-ms flag definition: packages/cli/src/commands/serve.ts:589type: number, description "ACP child request timeout ... Default: 10000".
  • Validation in packages/cli/src/serve/run-qwen-serve.ts:477–4122: MAX_TIMEOUT_MS = 2_147_483_647; isPositiveIntegerMs + assertTimerDelayInRange. 60 000 satisfies both.
  • No assertions in the test file reference ACP_INITIALIZE_TIMEOUT_MS; the value does not appear in the capabilities tag list (integration-tests/cli/qwen-serve-routes.test.ts:332+).
  • Product default (DEFAULT_INIT_TIMEOUT_MS = 10_000 in packages/acp-bridge/src/bridge.ts) is untouched.
  • CONTRIBUTING.md: no AI-review ban.

Not covered:

  • Working tree unavailable; local integration test run not possible. Scan tier does not require it — this change has no runtime logic, only a numeric flag value.

Cross-check: one existing review (qwen-code-ci-bot, COMMENTED) — zero findings, downgraded from Approve due to CI still running. No finding conflicts; CI state is outside this review's verdict per the platform adapter.

Reviewed with AI assistance.

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Summary

Package Lines Statements Functions Branches
CLI N/A% N/A% N/A% N/A%
Core N/A% N/A% N/A% N/A%
CLI Package - Full Text Report
CLI full-text-summary.txt not found at: coverage_artifact/cli/coverage/full-text-summary.txt
Core Package - Full Text Report
Core full-text-summary.txt not found at: coverage_artifact/core/coverage/full-text-summary.txt

For detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run.

@wenshao
wenshao added this pull request to the merge queue Sep 2, 2026
Merged via the queue into main with commit 8fde141 Sep 2, 2026
85 of 88 checks passed
@qwen-code-ci-bot

Copy link
Copy Markdown
Collaborator

Released in v0.23.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Main CI failed: E2E Tests on 76c32fd5ea4a

4 participants