ci(e2e): run Linux E2E with Vitest forks (#11268) - #11279
qwen-code-dev-bot wants to merge 5 commits into
Conversation
…etryable (#11268) Run 34083672277's sandbox:none shard 1/3 degraded under shared-host pressure instead of dying fast: it burned 40 of the 60 job minutes beside 14-18 minute sibling shards, then exited with every test green and no vitest FAIL line — the known transient class, but at a job- elapsed of ~2591s the budget gate had no shard-time left, so the retry added in #10355 never fired and the run filed a per-commit issue. Bound the first attempt with timeout(1) at 1500s: above the worst measured healthy shard (~21min under pressure) and still inside the 2100s gate, so a degraded attempt is declared dead while its retry is still reachable. The retry itself stays unbounded; the gate has already reserved its shard-time. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
Autofix E2E Report — Issue #11268DiagnosisIssue #11268 tracks run 34083672277 (main @ Evidence gathered from the GitHub API (job/step metadata) and local reproduction:
Root cause: the retry machinery covers a shard that dies fast under shared-host pressure, but not a shard that degrades slowly — one that crawls at 2-3x normal duration and dies late. A slow death lands past the budget gate and forfeits the retry that exists to absorb this failure class. Fix
Tests: the pin suite gains a witness that exactly the first attempt is bounded; the execution harness (which runs the step's real script under Verification
中文说明Autofix E2E 报告 — Issue #11268诊断Issue #11268 跟踪的是 main 分支上 通过 GitHub API（任务/步骤元数据）和本地复现收集到的证据：
根本原因：现有重试机制能覆盖在共享宿主机压力下快速死亡的分片，但覆盖不了缓慢劣化的分片——即以正常速度 2-3 倍爬行、很晚才死的分片。缓慢死亡会落在预算门槛之后，使为重试这一故障类别而生的机制彻底失效。 修复仅修改
测试：pin 套件新增了"只有首次尝试受限"的见证；执行级测试套件（在 验证
🧠 Handled by Qwen Code · model/模型 |
github-actions
Bot
added
the
review/self-reported
|
Thanks for the PR — the problem statement here is unusually well evidenced, and I verified it independently against the real run rather than taking it on faith. Template looks good ✓ — every required heading is present, the Tested-on table is filled in, and the Chinese translation is complete rather than abbreviated. Problem: real and confirmed. I pulled run Direction: aligned. A transient that turns into a red Size: not applicable. No core paths are touched ( Approach: the scope is right, but I have a substantive concern about the constant — details in the code review. The diff is minimal and carries no drive-by churn, and Risk: no elevated Stage 1e signals. One thing worth flagging up front because it bounds how much of this CI can attest to: Moving on to code review. 🔍 中文说明感谢贡献——这个 PR 的问题陈述证据相当充分，我也没有直接采信，而是独立核对了真实的运行数据。 模板完整 ✓——所有必需标题都在，测试平台表格已填写，中文说明也是完整翻译而非摘要。 问题：真实存在，已确认。 我按步骤拆解了运行 方向：一致。 瞬态故障变成 main 分支红色是真实成本，而仓库已有重试机制——本 PR 是在扩展它，而不是新增一套平行机制。 规模： 不适用。未触及核心路径（ 方案：范围是对的，但那个常量我有实质性疑虑——细节见代码审查。 diff 很小，没有夹带无关改动， 风险： 无 Stage 1e 升级信号。有一点值得先说明，因为它决定了 CI 到底能为本改动提供多少证据： 进入代码审查 🔍 — Qwen Code · qwen3.8-max-2026-09-02 Reviewed at |
Code reviewThe mechanism is right and the witnessing is genuinely good. I have one blocker, and it isn't about the code — it's about the constant. Blocker: 1500s sits below the measured healthy-shard maximum, so the bound kills green shardsThe justification in both the diff comment and the PR body is that "1500s clears the worst measured healthy shard (~21min under pressure)". I measured it rather than trusting the inherited figure, and it doesn't hold. I took the last 13
Three healthy shards that passed on their first and only attempt ran 26.2, 30.0 and 30.3 minutes. The cap would have killed all three. What that costs, using each leg's own measured setup — the gate admits the retry every time (elapsed ≈ 1709–1748s ≤ 2100s), and then the retry needs as long as the attempt that was just killed:
So "worst case one extra shard-time" understates it. Two of the three land within about a minute of The deeper problem is that there is no wall-clock cap that satisfies both constraints at the current settings:
That's a ~33-second window, with zero margin, derived from a 13-run sample whose true tail is higher. Picking 1500s lands 319s on the wrong side of it. I think the "~21min" figure you inherited from the gate comment describes a failed attempt rather than a healthy maximum — So this needs a design decision rather than a different constant: move the gate's 2100s threshold and/or the job's One accuracy correction (non-blocking)"every other leg of the run was green" isn't true for run What I checked and believe is correct — no need to re-litigate these
Test evidenceThis is an unattended CI run, so per the gate rules I did not build or execute anything from this PR. Everything below is read from the PR's own checks via the API at the reviewed commit.
The job that matters here is
The macOS and Windows Not verified, and it's the gap that matters: Sandboxed verification would settle this: 中文说明代码审查机制是对的，测试见证也做得扎实。我有一个阻塞项，而且不在代码本身——在那个常量上。 阻塞项：1500 秒低于实测的健康分片最大耗时，因此这个时限会杀掉本来绿色的分片diff 注释和 PR 正文的依据都是"1500 秒高于实测最慢健康分片（压力下约 21 分钟）"。我没有采信这个继承来的数字，而是实测了一遍，结论是它不成立。 我取了最近 13 次 代价如何，用每条分支自己实测的 setup 计算——门槛每次都会放行重试（elapsed 约 1709–1748 秒 ≤ 2100 秒），而重试需要的时间与被杀掉的尝试一样长：
所以"最坏情况多花一次分片时长"这个说法低估了。三条中有两条落在距 更深层的问题是：在当前配置下，不存在同时满足两个约束的墙钟上限：
这是一个约 33 秒宽的窗口，零余量，而且来自一个真实尾部更高的 13 次运行样本。选 1500 秒等于落在窗口错误一侧 319 秒。 我认为你从门槛注释继承的"约 21 分钟"描述的是失败尝试而非健康最大值—— 所以这里需要的是设计决策，而不是换一个常量：调整门槛的 2100 秒阈值和/或任务的 一处准确性更正（非阻塞）"其余分支也全部通过"对运行 我已核实且认为正确的部分——无需再讨论
测试证据这是无人值守的 CI 运行，因此按门禁规则我没有构建或执行本 PR 的任何代码。以下内容全部通过 API 从该 PR 自身的检查结果中读取，对应被审查的提交。 上表中真正关键的是
macOS 与 Windows 的 未验证，而且这是关键缺口： 沙箱验证可以解决这一点： — Qwen Code · qwen3.8-max-2026-09-02 Reviewed at |
|
Confidence: 2/5 — the mechanism is sound and the witnessing is better than most CI changes here, but the one number the whole change rests on is contradicted by this repo's own last 13 Stepping back: I wrote down what I'd do before reading the diff, and I landed on the same mechanism — bound the first attempt with Where I'd have done it differently is the constant. I'd have derived the cap from the measured shard-duration distribution instead of inheriting the gate comment's "~21min worst measured healthy shard". Measuring it is what breaks the PR: healthy shards that passed on their first and only attempt ran 26.2, 30.0 and 30.3 minutes in this window, so 1500s doesn't clear the distribution — it cuts through the middle of its upper half. And the two constraints (retry reachable ⇒ cap ≤ ~1852s; healthy shards survive ⇒ cap > 1819s) leave about 33 seconds of viable window, which tells me this isn't a wrong number so much as no number works at the current gate and job budget. The reason I'm not willing to let it land anyway, despite the mechanism being right: this failure is silent where it matters. A false-positive kill produces a job cancelled at 58 minutes of a 60-minute budget, which reads as exactly the shared-host flakiness the retry exists to absorb. Nobody triaging that red To be clear about what I'm not saying: the problem is real (I verified run What "fixed" looks like, concretely: the three numbers — Two things a maintainer should see, since I can't resolve either from the diff: moving Requesting changes on the cap calibration. 🙏 中文说明置信度：2/5 —— 机制是对的，测试见证也比这里多数 CI 改动做得好，但整个改动所依赖的那一个数字，被本仓库最近 13 次 退一步看：我在读 diff 之前先写下了自己会怎么做，结论是同一个机制——用 我会做得不一样的地方是那个常量。我会从实测的分片耗时分布推导上限，而不是继承门槛注释里的"压力下最慢健康分片约 21 分钟"。一实测就把这个 PR 推翻了：在这个窗口内，有健康分片在首次且唯一一次尝试中就跑完并通过，耗时分别为 26.2、30.0 和 30.3 分钟。所以 1500 秒并没有高于分布——它切进了分布上半段的中间。而两个约束（重试可达 ⇒ cap ≤ 约 1852 秒；健康分片存活 ⇒ cap > 1819 秒）只留下约 33 秒的可行窗口，这说明问题不在于数字选错了，而在于在当前门槛与任务预算下没有数字可行。 我不愿意就这样让它合并的原因（尽管机制是对的）：这个失败在关键处是静默的。一次误杀会产生一个在 60 分钟预算中第 58 分钟被取消的任务，看起来完全就是重试机制本要吸收的共享宿主机抖动。排查那次红色 需要说清楚我没有在讲什么：问题是真实的（我在步骤级核实了运行 具体说"修好"是什么样子： 有两点需要维护者过目，因为我都无法从 diff 中解决：调整 就上限标定问题请求修改。🙏 — Qwen Code · qwen3.8-max-2026-09-02 Reviewed at |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Requesting changes on one point: the 1500s cap is calibrated below this repo's measured healthy-shard maximum.
Across the last 13 main runs of e2e.yml, three sandbox:none shards passed on their first and only attempt at 26.2, 30.0 and 30.3 minutes — the cap would have killed all three, then re-run them into a job budget they only just fit (53 and 72 seconds of margin in two of the three cases).
The full measurement table, the per-leg arithmetic, and the parts I verified as correct are in the code-review comment above; the reflection comment has what a fixed calibration would need to satisfy.
The mechanism, the export -f witness, and the problem diagnosis are all sound — this needs the cap and the gate's 2100s threshold re-derived together, not a rethink. 🙏
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Partially reviewed — gaps disclosed.
Not explored to full depth (tool budget reached): "agent 3b": could not run the two touched test files ( npx vitest run --config ./scripts/tests/vitest.config.ts … fails in this review worktree with Cannot find package '….
Not reviewed: reverse audit — stopped before round 1 by the review time budget.
Not linted (tool limitation, not a blocker): .github/workflows/e2e.yml — actionlint embedded-shell source mapping is not yet supported.
中文说明
仅完成部分审查，审查缺口已披露。
未探索到全部深度（达到工具调用预算）："agent 3b"：could not run the two touched test files ( npx vitest run --config ./scripts/tests/vitest.config.ts … fails in this review worktree with Cannot find package '…。
未审查：反向审计——评审时间预算不足，未能开始第 1 轮。
未检查（工具限制，非阻断）：.github/workflows/e2e.yml——actionlint 对 workflow 内嵌 shell 的源映射尚未支持。
— qwen3.8-max via Qwen Code /review (v0.23.0)
| # group, and --kill-after promotes a wedged one to KILL. The | ||
| # retry stays unbounded: the gate below has already reserved its | ||
| # shard-time. | ||
| timeout --kill-after=30 1500 bash -c run_shard || { |
There was a problem hiding this comment.
[Critical] R1-1: [certifies-falsely] [regression] The 1500s cap sits below this leg's measured healthy single-attempt maximum, so it kills green shards — and the constraint set turns out to be empty, not merely mis-tuned.
We measured the healthy baseline from GitHub's own step timings and vitest's own summaries on main runs whose logs still echo the pre-change script (grep -c 'timeout --kill-after' returns 0), so these are true baselines rather than post-change artefacts. Three of the five green single-attempt legs measured exceed 1500s, by 286s, 299s and 70s:
| Run / job | Leg, host | setup | test step | vitest summary | conclusion |
|---|---|---|---|---|---|
| 34066983358 / 101584460519 | 3/3, hk3-13 | 179s | 1819s | Test Files 24 passed (24), Duration 1816.21s, zero ##[warning] so no retry fired |
success |
| 34030617708 / 101483905505 | 3/3, hk3-11 | 218s | 1799s | Test Files 25 passed (25) |
success |
| 34030617708 / 101483905455 | 1/3, hk3-9 | 203s | 1570s | Test Files 20 passed | 5 skipped (25) |
success |
Each of those would be SIGTERM'd at 1500s and SIGKILL'd at 1530s, between 70s and 299s before it reported green. The retry the cap exists to unlock then does not fit either: 218 + 1530 = 1748s at the gate, which is inside 2100s so the retry is admitted, and that retry needs the measured 1799s — a job total of 3547s against the 3600s budget, leaving 53s. For job 101584460519 the same arithmetic gives 179 + 1530 + 1816 = 3525s. Run 34030617708 shard 2/3 (job 101483905502) shows what that boundary actually does: cancelled at 3622s, past timeout-minutes: 60.
The reason a different constant will not fix this is that the job budget, not the gate, is the binding constraint. Keeping a killed attempt's retry inside the job needs setup + cap + kill_after + retry <= 3600, i.e. cap <= 3600 - 218 - 30 - 1819 = 1533s, while sparing healthy shards needs cap > 1819s. Two ~1800s attempts plus setup simply do not fit one 60-minute job (218 + 1819 + 1819 = 3856s). So on exactly the slow-host population this bound targets, no cap value both spares healthy shards and lets the bound-kill's retry finish.
This independently corroborates the blocker already raised in the triage review on this pull request — our measurement agrees with it (1816s from the vitest summary, 1819s of step wall clock) — and adds the empty-constraint-set result above, which is why the fix has to move more than one number.
Witness:
base arm proven first — all four measured logs echo the PRE-change script:
grep -c 'timeout --kill-after' = 0 ; '457: run_shard || {' ; no 'export -f run_shard'
run 34066983358 is event=push head_branch=main head_sha=7567824d4c
34066983358/101584460519 setup 179s step 1819s 'Test Files 24 passed (24)'
'Duration 1816.21s' 0 warnings success
34030617708/101483905505 setup 218s step 1799s 'Test Files 25 passed (25)' success
34030617708/101483905455 setup 203s step 1570s success
=> 3 of 5 measured green single-attempt legs exceed the 1500s cap
cap semantics probe:
timeout --kill-after=30 2 bash -c slow.sh (5s script ending in
echo "SHARD REPORTED GREEN")
-> rc=124 wall_ms=2002, NO green line — the attempt dies before it reports
retry-fits: 218 + 1530 = 1748 <= 2100 (admitted) + 1799 retry = 3547s of 3600s
boundary observed: job 101483905502 CANCELLED 12:09:49 -> 13:10:11 = 3622s
Re-derive the cap, the gate's 2100s threshold and timeout-minutes together, and check the result against 3600s rather than only against the 2100s gate: either raise timeout-minutes, bound the retry as well, or bound on a signal other than total wall clock. Whichever pair is chosen, please record the measurement behind it in the comment — the "~21min" figure cited there describes a failed attempt (run 34081391271's first attempt died at elapsed 1417s, about 20.3 minutes of attempt time), not a healthy maximum, which measures 30.3 minutes.
Any re-derived value rests on two existing facts that must not be violated: timeout-minutes: 60 at .github/workflows/e2e.yml:112, pinned by expect(yml.jobs['e2e-test-linux']['timeout-minutes']).toBe(60) in scripts/tests/e2e-workflow.test.js, and if (( elapsed > 2100 )); then at :389, whose own derivation comment reads "3600s minus a 25-minute reserve — the worst measured shard is ~21min under shared-host pressure — leaves 2100s". expect(runStep.run.match(/bash -c run_shard/g)).toHaveLength(1) also pins that only one attempt is wrapped.
scripts/tests/e2e-shard-retry.test.js asserts expect(timeoutDurations).toEqual(['1500']) in four cases and scripts/tests/e2e-workflow.test.js asserts toContain('timeout --kill-after=30 1500 bash -c run_shard || {'), so both go red when the constant moves — please update them in the same change. Worth saying plainly that they pin the value, not its correctness against measurement: nothing in the harness can, since timeout and the clock are both stubbed.
中文说明
1500 秒的上限低于本分支实测的健康单次尝试最大值，因此它会杀掉本来绿色的分片——而且约束集合是空的，并非只是调错了数值。
我们从 GitHub 自己的步骤计时与 vitest 自己的汇总中取得健康基线，取样运行的日志仍然是改动前的脚本（grep -c 'timeout --kill-after' 返回 0），所以它们是真实基线。实测五条绿色单次尝试分支中，有三条超过 1500 秒，分别超出 286 秒、299 秒和 70 秒：
| 运行 / 任务 | 分支、宿主 | setup | 测试步骤 | vitest 汇总 | 结论 |
|---|---|---|---|---|---|
| 34066983358 / 101584460519 | 3/3，hk3-13 | 179s | 1819s | Test Files 24 passed (24)、Duration 1816.21s，零条 ##[warning] 即未触发重试 |
success |
| 34030617708 / 101483905505 | 3/3，hk3-11 | 218s | 1799s | Test Files 25 passed (25) |
success |
| 34030617708 / 101483905455 | 1/3，hk3-9 | 203s | 1570s | Test Files 20 passed | 5 skipped (25) |
success |
这三条都会在 1500 秒被 SIGTERM、1530 秒被 SIGKILL，比它们报告绿色早 70 到 299 秒。而这个上限本想换来的重试同样放不下：门槛处 218 + 1530 = 1748 秒，在 2100 秒之内所以重试被放行，但重试需要实测的 1799 秒——任务合计 3547 秒，而预算是 3600 秒，只剩 53 秒。对 101584460519 而言同样的算术是 179 + 1530 + 1816 = 3525 秒。运行 34030617708 的 shard 2/3（任务 101483905502）演示了这个边界会发生什么：在 3622 秒被取消，超过了 timeout-minutes: 60。
换一个常量修不好这个问题，原因在于真正起约束作用的是任务预算而不是门槛。要让被杀尝试的重试留在任务之内，需要 setup + cap + kill_after + retry <= 3600，即 cap <= 3600 - 218 - 30 - 1819 = 1533 秒；而要让健康分片不被杀，需要 cap > 1819 秒。两次约 1800 秒的尝试加上 setup 根本放不进一个 60 分钟的任务（218 + 1819 + 1819 = 3856 秒）。所以在这个上限所要针对的慢宿主机群体上，不存在既能放过健康分片、又能让被杀尝试的重试跑完的上限值。
这一结论独立地印证了本 PR 上 triage 评审已提出的阻塞项——我们的实测与之一致（vitest 汇总 1816 秒、步骤墙钟 1819 秒）——并补充了上面的"约束集合为空"这一结果，这也是为什么修复必须同时改动不止一个数字。
见证：
先证明基线臂 —— 四条实测日志都是改动前的脚本：
grep -c 'timeout --kill-after' = 0 ；'457: run_shard || {' ；没有 'export -f run_shard'
运行 34066983358 为 event=push head_branch=main head_sha=7567824d4c
34066983358/101584460519 setup 179s step 1819s 'Test Files 24 passed (24)'
'Duration 1816.21s' 0 warnings success
34030617708/101483905505 setup 218s step 1799s 'Test Files 25 passed (25)' success
34030617708/101483905455 setup 203s step 1570s success
=> 实测 5 条绿色单次尝试分支中有 3 条超过 1500 秒上限
上限语义探针：
timeout --kill-after=30 2 bash -c slow.sh （5 秒脚本，结尾为
echo "SHARD REPORTED GREEN"）
-> rc=124 wall_ms=2002，没有绿色输出行 —— 尝试在报告之前就被杀
重试是否放得下：218 + 1530 = 1748 <= 2100（放行）+ 1799 重试 = 3547s / 3600s
实测边界：任务 101483905502 被取消，12:09:49 -> 13:10:11 = 3622s
请把上限、门槛的 2100 秒阈值和 timeout-minutes 三者一起重新推导，并用 3600 秒而不只是 2100 秒门槛来检验结果：要么提高 timeout-minutes，要么也给重试设上限，要么改用总墙钟以外的信号来设限。无论选哪一组，请在注释中记录其依据的实测数据——注释里引用的"约 21 分钟"描述的是一次失败的尝试（运行 34081391271 的首次尝试死在 elapsed 1417 秒，约 20.3 分钟尝试时长），而不是健康最大值；实测健康最大值为 30.3 分钟。
任何重新推导出的值都依赖两条不可违反的既有事实：.github/workflows/e2e.yml:112 的 timeout-minutes: 60（由 scripts/tests/e2e-workflow.test.js 中的 expect(yml.jobs['e2e-test-linux']['timeout-minutes']).toBe(60) 钉住），以及 :389 的 if (( elapsed > 2100 )); then（其自身的推导注释为"3600s minus a 25-minute reserve — the worst measured shard is ~21min under shared-host pressure — leaves 2100s"）。expect(runStep.run.match(/bash -c run_shard/g)).toHaveLength(1) 还钉住了只包裹一次尝试。
scripts/tests/e2e-shard-retry.test.js 在四个用例中断言 expect(timeoutDurations).toEqual(['1500'])，scripts/tests/e2e-workflow.test.js 断言 toContain('timeout --kill-after=30 1500 bash -c run_shard || {')，所以常量一变两者都会变红——请在同一次改动中更新它们。需要坦白说明的是：它们钉住的是数值，而不是该数值相对实测的正确性；测试套件无法做到后者，因为 timeout 和时钟都被打了桩。
— qwen3.8-max via Qwen Code /review (v0.23.0)
| # The first attempt is wall-clock bounded: the same pressure can | ||
| # degrade a shard slowly instead of killing it — run 34083672277's | ||
| # shard 1/3 burned 40 of the 60 job minutes beside 14-18 minute | ||
| # siblings and died with every test green and the retry budget |
There was a problem hiding this comment.
[Suggestion] R1-2: The witness this comment records for its own motivating run is not what that run's log shows — the shard did not die "with every test green", it reported two named test failures.
We re-fetched the exact job the linked issue names (job 101627904968 of run 34083672277, E2E Test (Linux) - sandbox:none - shard 1/3 on ecs-qwen-hk3-15). Its single vitest summary reads Test Files 1 failed | 20 passed | 5 skipped (26), Tests 2 failed | 155 passed | 19 skipped (176), Duration 2413.51s, with two named failures in interactive/context-compress-interactive.test.ts: should trigger chat compression with /compress command and should forward /compress instructions through to the side-query, both AssertionError: chat_compression telemetry event was not found: expected false to be true, at lines 107 and 203 after a 90000ms wait. grep -c '##[warning]' on that job returns 0, so it was one attempt, not two. The rest of this comment's witness is roughly right — the job ran 05:01:39 to 05:45:10, about 43.5 of the 60 minutes, and the gate refused at 2590s — though the siblings measured 16.3 and 20.6 minutes rather than "14-18 minute".
Two concrete costs follow. The linked issue closes on the "all-green silent death" class while the defect actually observed at that commit — two chat_compression telemetry assertions failing — is never triaged; if those are not host-pressure flakes, main goes red again and the next per-commit issue gets filed against a different commit with the same unaddressed cause. And because that shard's summary only prints at 2413s, the new 1500s kill destroys the only output that names the failing tests for this shard's slow shape: after this change the first attempt produces exit 124 and no report at all, which is strictly less diagnostic than the pre-change red at 2590s. The same premise is repeated in the new test's comment, "timeout(1) kills the degraded first attempt (exit 124, the shard never reporting)". One corroborating detail: the pull request's own local figure of "157 passed, 19 skipped, 0 failed" is exactly CI's 155 passed + 2 failed.
Witness:
gh api repos/QwenLM/qwen-code/actions/jobs/101627904968/logs (1,138,140 bytes)
Test Files 1 failed | 20 passed | 5 skipped (26)
Tests 2 failed | 155 passed | 19 skipped (176)
Duration 2413.51s
FAIL interactive/context-compress-interactive.test.ts > Interactive Mode >
should trigger chat compression with /compress command
FAIL interactive/context-compress-interactive.test.ts > Interactive Mode >
should forward /compress instructions through to the side-query
AssertionError: chat_compression telemetry event was not found:
expected false to be true (:107:76 and :203:76, after a 90000ms wait)
##[error]sandbox:none shard failed on ecs-qwen-hk3-15 after 2590s of the
3600s job budget — not enough left for a retry
grep -c '##[warning]' = 0 => one attempt, no retry fired
job 05:01:39 -> 05:45:10 = 43.5 min of 60 ; siblings 16.3 min and 20.6 min
Restate the witness as the log shows it — the attempt ran 2413s under pressure and reported two failures in interactive/context-compress-interactive.test.ts (chat_compression telemetry event was not found), reaching the gate at elapsed 2590s, past the 2100s threshold — and correct both "every test green" and the sibling range. Then either triage those two assertions or say explicitly in the description that they are being left to the retry, so the issue does not close on a failure class the run does not exhibit.
We are filing this as a Suggestion rather than a blocker deliberately: the code does nothing wrong here, an added comment misdescribes its own evidence, and the behavioural consequence of that misdescription — a cap derived from a false healthy maximum — is already carried by the Critical above, so blocking twice on one defect would not help. The triage review on this pull request labelled its own analogous accuracy correction non-blocking as well.
中文说明
这段注释为其动机运行所记录的见证，与该运行日志的实际内容不符——那个分片并不是"所有测试都绿色"地死掉的，它报告了两个具名的测试失败。
我们重新拉取了关联 issue 所指的那个任务（运行 34083672277 的任务 101627904968，ecs-qwen-hk3-15 上的 E2E Test (Linux) - sandbox:none - shard 1/3）。它唯一的 vitest 汇总是 Test Files 1 failed | 20 passed | 5 skipped (26)、Tests 2 failed | 155 passed | 19 skipped (176)、Duration 2413.51s，其中两个具名失败位于 interactive/context-compress-interactive.test.ts：should trigger chat compression with /compress command 与 should forward /compress instructions through to the side-query，都是 AssertionError: chat_compression telemetry event was not found: expected false to be true，分别在 107 行和 203 行，等待 90000ms 之后失败。对该任务 grep -c '##[warning]' 返回 0，所以只有一次尝试，没有重试。这段注释其余的见证大致正确——任务从 05:01:39 跑到 05:45:10，约占 60 分钟中的 43.5 分钟，门槛在 2590 秒处拒绝——不过兄弟分片实测为 16.3 和 20.6 分钟，而不是"14-18 分钟"。
由此产生两项具体代价。其一，关联 issue 会以"全绿静默死亡"这一类别被关闭，而在该提交上实际观测到的缺陷——两个 chat_compression 遥测断言失败——从未被分诊；如果它们并非宿主机压力导致的抖动，main 会再次变红，而下一份按提交建档的 issue 会被提到另一个提交上，同样的根因仍未处理。其二，由于该分片的汇总要到 2413 秒才输出，新的 1500 秒杀会摧毁唯一能指明失败测试的输出：改动之后首次尝试只会产生退出码 124 而完全没有报告，这比改动前 2590 秒处的红色诊断信息更少。同样的前提也出现在新增测试的注释里——"timeout(1) kills the degraded first attempt (exit 124, the shard never reporting)"。一个佐证细节：PR 自己给出的本地数据"157 passed, 19 skipped, 0 failed"恰好等于 CI 的 155 passed + 2 failed。
见证：
gh api repos/QwenLM/qwen-code/actions/jobs/101627904968/logs (1,138,140 字节)
Test Files 1 failed | 20 passed | 5 skipped (26)
Tests 2 failed | 155 passed | 19 skipped (176)
Duration 2413.51s
FAIL interactive/context-compress-interactive.test.ts > Interactive Mode >
should trigger chat compression with /compress command
FAIL interactive/context-compress-interactive.test.ts > Interactive Mode >
should forward /compress instructions through to the side-query
AssertionError: chat_compression telemetry event was not found:
expected false to be true (:107:76 与 :203:76，等待 90000ms 后)
##[error]sandbox:none shard failed on ecs-qwen-hk3-15 after 2590s of the
3600s job budget — not enough left for a retry
grep -c '##[warning]' = 0 => 一次尝试，未触发重试
任务 05:01:39 -> 05:45:10 = 60 分钟中的 43.5 分钟；兄弟分片 16.3 与 20.6 分钟
请按日志的实际内容重述这段见证——该次尝试在压力下运行了 2413 秒，并在 interactive/context-compress-interactive.test.ts 中报告了两个失败（chat_compression telemetry event was not found），到达门槛时 elapsed 为 2590 秒，超过 2100 秒阈值——同时更正"every test green"和兄弟分片的时长区间。然后要么分诊那两个断言，要么在描述中明确说明把它们交给重试处理，这样 issue 才不会以该运行并不具备的失败类别被关闭。
我们刻意把这一条定为 Suggestion 而不是阻塞项：这里代码本身没有做错什么，是一段新增注释误述了自己的证据；而这一误述带来的行为后果——上限是基于一个错误的健康最大值推导出来的——已经由上面的 Critical 承载，就同一个缺陷重复阻塞两次并无帮助。本 PR 上的 triage 评审也把自己类似的准确性更正标为非阻塞。
— qwen3.8-max via Qwen Code /review (v0.23.0)
| # siblings and died with every test green and the retry budget | ||
| # already spent, which forfeits the retry this block exists to | ||
| # enable. 1500s clears the worst measured healthy shard (~21min | ||
| # under pressure, per the gate arithmetic below) and still lands |
There was a problem hiding this comment.
[Suggestion] R1-3: This claim is measured from a different clock than the gate it refers to, and nothing in the workflow or either test file relates the two constants — so the guarantee reads as unconditional while it is actually conditional, and a future edit to either number ships green.
The cap runs from attempt start, while the gate at :389 runs from job start: E2E_JOB_START_EPOCH is written by the job's first step at :143, before checkout, dependency install and artifact unpack. A timeout-killed attempt therefore reaches the gate at setup + 1500 + 30, which lands inside 2100s only while pre-shard setup stays under about 570s. We drove the step extracted from this commit under the harness's own stubs and confirmed the other branch: at a large pre-shard elapsed the bound-killed attempt lands past the gate and the retry never happens, which is precisely the outcome this hunk was added to prevent.
To be clear about what is and is not broken today, because it changes the urgency: this is a latent fragility, not a current failure. Setup measured 98s, 109s, 173s, 179s, 203s and 218s across six legs on six different pool hosts, so setup + 1500 + 30 is 1628-1748s and sits comfortably inside the gate. The "~29 of those minutes" setup figure at :139-141 describes a job shape that no longer exists — on old run 33293739505 job 99213809318, job start 05:43:01 to test step 06:13:26 is 1825s of setup, of which an in-job Build project was 1495s and Bundle CLI for E2E tests 35s. Today's leg sets QWEN_SKIP_PREPARE: '1' and instead downloads and unpacks the build job's artifact in 27s and 17s, so heavy setup cannot recur without a structural change to the job. What remains is the missing coupling witness: 1500 at :382 and 2100 at :389 are independent, the tests pin each separately, and a maintainer re-tuning either one reads a green suite as proof of a guarantee that only holds below roughly 570s of setup.
Witness:
extracted step at HEAD, harness stubs (authority: qwen review extract-step)
large pre-shard elapsed, timeoutExit=124
-> exitCode=1 npmCalls=0 durations=[1500]
::error::sandbox:none shard failed on ... after 2400s of the 3600s
job budget — not enough left for a retry
setup distribution measured over six legs on six hosts:
98s, 109s, 173s, 179s, 203s, 218s => setup + 1500 + 30 = 1628-1748s <= 2100s
stale premise: run 33293739505 job 99213809318
job start 05:43:01 -> test step 06:13:26 = 1825s setup
of which in-job 'Build project' 1495s, 'Bundle CLI for E2E tests' 35s
today: QWEN_SKIP_PREPARE='1', 'Download build artifact' 27s + 'Unpack' 17s
Either derive the first-attempt bound from the same clock the gate reads — read elapsed before the attempt and bound with what the gate can still admit, failing fast through the existing ::error:: path when that falls below one shard-time — or keep the fixed 1500s and state its precondition here, that a killed attempt lands inside the gate only while pre-shard setup stays under about 570s. Please also refresh the "~29 of those minutes" comment at :139-141, which our measurement shows describes a job structure this leg no longer has, and which is what makes the current wording read as an unconditional guarantee.
A derived bound has to respect the existing pins: if (( elapsed > 2100 )); then at :389 and timeout-minutes: 60 at :112; E2E_JOB_START_EPOCH is written by the job's first step at :143, so elapsed includes checkout, install and unpack and cannot be treated as step-local; and expect(runStep.run.match(/bash -c run_shard/g)).toHaveLength(1) together with the end-anchored /run_shard\s*\n\s*\}\s*\n\s*fi\s*$/ in scripts/tests/e2e-workflow.test.js mean a derived cap must keep a single bounded call site and keep the retry as the group's last command.
For the witness, all four existing cases in scripts/tests/e2e-shard-retry.test.js assert the constant expect(timeoutDurations).toEqual(['1500']), so a derived bound currently has no witness at all — please add a case that varies elapsedSeconds and asserts the recorded duration tracks the remaining budget rather than the constant, and confirm it reds when the derivation is reverted to a flat 1500. For the comment-only alternative, a case at a heavy-setup elapsed with timeoutExit: 124 asserting the gate refuses (npmCalls 0, ::error::, non-zero exit) would pin in code that a bound-kill does not always reach the retry.
中文说明
这句结论所用的时钟与它所指门槛的时钟并不是同一个，而工作流和两个测试文件中都没有任何东西把这两个常量关联起来——于是这个保证读起来是无条件的，实际上却是有条件的，并且将来改动其中任一数字都会在套件全绿的情况下合入。
上限从尝试开始计时，而 :389 的门槛从任务开始计时：E2E_JOB_START_EPOCH 由任务的第一步在 :143 写入，早于检出、依赖安装与产物解包。因此被时限杀掉的尝试到达门槛时是 setup + 1500 + 30，只有在分片前 setup 低于约 570 秒时才落在 2100 秒之内。我们用测试套件自己的桩驱动了从本提交提取出的步骤脚本，确认了另一分支的行为：当分片前 elapsed 较大时，被上限杀掉的尝试会落在门槛之外，重试根本不会发生——而这正是本段改动想要避免的结果。
需要说清楚今天什么是坏的、什么不是坏的，因为这决定了紧迫程度：这是一个潜在脆弱点，不是当前的故障。在六个不同池宿主机的六条分支上实测 setup 为 98 秒、109 秒、173 秒、179 秒、203 秒和 218 秒，所以 setup + 1500 + 30 为 1628-1748 秒，宽裕地落在门槛之内。:139-141 处"约 29 分钟"的 setup 数字描述的是一种已不存在的任务形态——在旧运行 33293739505 的任务 99213809318 上，任务开始 05:43:01 到测试步骤 06:13:26 是 1825 秒 setup，其中任务内 Build project 占 1495 秒、Bundle CLI for E2E tests 占 35 秒。今天的分支设置了 QWEN_SKIP_PREPARE: '1'，改为下载并解包 build 任务的产物，分别耗时 27 秒和 17 秒，所以除非任务结构发生变化，重度 setup 不会重现。真正剩下的是缺失的耦合见证：:382 的 1500 与 :389 的 2100 彼此独立，测试分别钉住二者，于是维护者重新调整其中任何一个时，会把全绿的套件当成一个保证的证明，而该保证只在 setup 低于约 570 秒时成立。
见证：
从 HEAD 提取的步骤脚本，使用套件桩（依据：qwen review extract-step）
较大的分片前 elapsed，timeoutExit=124
-> exitCode=1 npmCalls=0 durations=[1500]
::error::sandbox:none shard failed on ... after 2400s of the 3600s
job budget — not enough left for a retry
六个宿主机上六条分支实测的 setup 分布：
98s、109s、173s、179s、203s、218s => setup + 1500 + 30 = 1628-1748s <= 2100s
过期前提：运行 33293739505 任务 99213809318
任务开始 05:43:01 -> 测试步骤 06:13:26 = 1825s setup
其中任务内 'Build project' 1495s，'Bundle CLI for E2E tests' 35s
今天：QWEN_SKIP_PREPARE='1'，'Download build artifact' 27s + 'Unpack' 17s
两种修法任选其一：要么让首次尝试的上限从门槛所读的同一个时钟推导——在尝试前读取 elapsed，并用门槛还能容纳的余量作为上限，当该余量低于一个分片时长时直接走既有的 ::error:: 快速失败路径；要么保留固定的 1500 秒，并在此处写明其前提条件，即只有分片前 setup 低于约 570 秒时，被杀的尝试才会落在门槛之内。同时请更新 :139-141 处"约 29 分钟"的注释——我们的实测表明它描述的是本分支已不再具备的任务结构，而正是它让当前的措辞读起来像无条件保证。
推导出的上限必须尊重既有的钉住项：:389 的 if (( elapsed > 2100 )); then 与 :112 的 timeout-minutes: 60；E2E_JOB_START_EPOCH 由任务第一步在 :143 写入，因此 elapsed 包含检出、安装与解包，不能当作步骤内时间处理；scripts/tests/e2e-workflow.test.js 中的 expect(runStep.run.match(/bash -c run_shard/g)).toHaveLength(1) 以及结尾锚定的 /run_shard\s*\n\s*\}\s*\n\s*fi\s*$/ 意味着推导式上限必须保持单一受限调用点，并保持重试为该命令组的最后一条命令。
关于见证：scripts/tests/e2e-shard-retry.test.js 现有的四个用例都断言常量 expect(timeoutDurations).toEqual(['1500'])，所以推导式上限目前完全没有见证——请新增一个用例，改变 elapsedSeconds 并断言记录到的时长跟随剩余预算而非常量，并确认当推导被改回固定 1500 时该用例变红。若选择只改注释的方案，则可以新增一个在重度 setup elapsed 下带 timeoutExit: 124 的用例，断言门槛拒绝（npmCalls 为 0、出现 ::error::、退出码非零），从而在代码中钉住"被上限杀掉的尝试并不总能到达重试"这一事实。
— qwen3.8-max via Qwen Code /review (v0.23.0)
| # under pressure, per the gate arithmetic below) and still lands | ||
| # a timeout-killed attempt inside the 2100s gate. Without | ||
| # --foreground, timeout signals the attempt's whole process | ||
| # group, and --kill-after promotes a wedged one to KILL. The |
There was a problem hiding this comment.
[Suggestion] R1-4: This change introduces a second, distinct cause of first-attempt death — our own bound — and routes it through the one message that names the other cause, so a bound-kill is indistinguishable in the job log from a genuine shard failure and the cap can never be diagnosed or counted from CI data.
The || { group reads only elapsed; the attempt's exit status is never captured. GNU timeout is silent by default (--verbose only exists from coreutils 9.0), so a bound-kill leaves no marker of its own — the shard's output simply stops, with every test green and no vitest FAIL line, which is verbatim the signature this block's own comment uses for the transient class. The step then prints ::warning::... retrying once (transient shared-host pressure class), asserting the wrong cause, and the gate-refusal ::error:: likewise says "shard failed" without saying the bound killed it.
We drove the step extracted from this commit twice, at the same elapsed, with different causes, and diffed the markers: they are identical. That matters concretely, because per the Critical above healthy shards are being killed 70-299s short of green — and every such occurrence will be diagnosed as shared-host pressure, so the cap is never revisited and no count of bound-kills can ever accumulate from run logs. That is a real loss here, since every other number in this block cites the run IDs it came from. The same message now also covers a deterministic product hang: killed at 1500s, retried, hangs again, cancelled by timeout-minutes, with every line the triager reads saying "transient pool pressure" — so a product bug gets filed as a runner flake.
There is an ordering trap in the fix worth flagging, because it fails silently: the capture must be the group's first statement, since elapsed=$(( $(date +%s) - ... )) at :383 is a command substitution that overwrites $?.
Witness:
two drives of the extracted step at HEAD, same elapsed, different cause:
bound-kill (timeoutExit=124):
exitCode=0 npmCalls=1 durations=[1500]
::warning::sandbox:none shard failed on ... after 1200s;
retrying once (transient shared-host pressure class)
genuine failure (npm exit 1):
exitCode=0 npmCalls=2 durations=[1500]
::warning::sandbox:none shard failed on ... after 1200s;
retrying once (transient shared-host pressure class)
diff of the two marker sets -> IDENTICAL MARKERS
grep -icE 'timeout|124|signal|kill|bound' on the full bound-kill output -> 0
timeout is silent by default:
timeout 1 sleep 3 -> rc=124 stdout=[] stderr=[]
(coreutils 9.4 does have -v, --verbose "diagnose to stderr any signal sent
upon timeout")
$? clobbering, probed:
bash -c 'false || { elapsed=$(( $(date +%s) - 0 )); echo "$?"; }' -> 0
bash -c 'false || { rc=$?; elapsed=$(( $(date +%s) - 0 )); echo "$rc"; }' -> 1
Capture the status as the group's first statement and name the bound when the bound is what fired, keeping the existing message for every other status:
timeout --kill-after=30 1500 bash -c run_shard || {
rc=$?
elapsed=$(( $(date +%s) - ${E2E_JOB_START_EPOCH:-0} ))
# ... budget gate unchanged, but interpolate the status into ::error:: too ...
if (( rc == 124 || rc == 137 )); then
echo "::warning::sandbox:none shard hit its 1500s first-attempt bound on ${RUNNER_NAME:-this runner} after ${elapsed}s; retrying once"
else
echo "::warning::sandbox:none shard failed on ${RUNNER_NAME:-this runner} after ${elapsed}s (attempt exit ${rc}); retrying once (transient shared-host pressure class)"
fi
run_shard
}The step runs under GitHub's default Linux shell bash -e {0}, pinned by expect(runStep.shell).toBeUndefined() plus the absent defaults: blocks in scripts/tests/e2e-workflow.test.js and by the harness's execFileSync('bash', ['-e', scriptFile]) — so the status has to be captured inside the || { ... } group as shown; a bare timeout ...; rc=$? aborts the step under -e. expect(runStep.run.match(/run_shard/g)).toHaveLength(4) also means the added capture must not introduce another run_shard token, and the end-anchored /run_shard\s*\n\s*\}\s*\n\s*fi\s*$/ requires the retry to stay the group's last command.
Please extend it('retries a first attempt killed by its own wall-clock bound') in scripts/tests/e2e-shard-retry.test.js — it already drives timeoutExit: 124 but asserts only the generic expect(output).toContain('::warning::'), which the ordinary failure path satisfies too — so that it asserts the bound-specific text, and add the absence assertion to the failCalls: '1' probes; then remove the rc capture and confirm that test goes red, and move the capture after the elapsed= assignment and confirm it goes red again logging exit 0.
中文说明
这次改动引入了第二种、性质不同的首次尝试死因——我们自己的上限——却把它交给了那条专门指认另一种死因的消息，因此在任务日志中，被上限杀掉与分片真正失败完全无法区分，而这个上限也永远无法从 CI 数据中被诊断或统计。
|| { 命令组只读取 elapsed，从未捕获尝试的退出状态。GNU timeout 默认静默（--verbose 从 coreutils 9.0 才有），所以被上限杀掉不会留下任何自身标记——分片输出就此停止，所有测试绿色、没有 vitest FAIL 行，而这恰好就是本段注释自己用来描述瞬态类别的特征。随后步骤打印 ::warning::... retrying once (transient shared-host pressure class)，断言了错误的成因；门槛拒绝时的 ::error:: 同样只说"shard failed"，不说是上限杀掉的。
我们从本提交提取步骤脚本，在相同 elapsed 下用两种不同成因各驱动一次并对比标记：完全相同。这一点有具体影响，因为依据上面的 Critical，健康分片正在距离报告绿色还差 70 到 299 秒时被杀掉——而每一次这样的发生都会被诊断为共享宿主机压力，于是这个上限永远不会被重新审视，运行日志中也永远无法累积出被上限杀掉的次数。这在本处是实打实的损失，因为这段注释里其他每个数字都引用了它们的来源运行 ID。同一条消息现在还会覆盖确定性的产品卡死：在 1500 秒被杀、重试、再次卡死、被 timeout-minutes 取消，而分诊者读到的每一行都写着"瞬态池压力"——于是一个产品缺陷被当作运行器抖动归档。
修复中有一个值得指出的顺序陷阱，因为它会静默失效：状态捕获必须是命令组的第一条语句，因为 :383 的 elapsed=$(( $(date +%s) - ... )) 是一次命令替换，会覆盖 $?。
见证：
从 HEAD 提取的步骤脚本，相同 elapsed、两种成因各驱动一次：
被上限杀掉（timeoutExit=124）：
exitCode=0 npmCalls=1 durations=[1500]
::warning::sandbox:none shard failed on ... after 1200s;
retrying once (transient shared-host pressure class)
真实失败（npm 退出 1）：
exitCode=0 npmCalls=2 durations=[1500]
::warning::sandbox:none shard failed on ... after 1200s;
retrying once (transient shared-host pressure class)
两组标记对比 -> 完全相同
对被上限杀掉的完整输出执行 grep -icE 'timeout|124|signal|kill|bound' -> 0
timeout 默认静默：
timeout 1 sleep 3 -> rc=124 stdout=[] stderr=[]
（coreutils 9.4 确实有 -v, --verbose "diagnose to stderr any signal sent
upon timeout"）
$? 被覆盖，已探针验证：
bash -c 'false || { elapsed=$(( $(date +%s) - 0 )); echo "$?"; }' -> 0
bash -c 'false || { rc=$?; elapsed=$(( $(date +%s) - 0 )); echo "$rc"; }' -> 1
请把状态捕获作为命令组的第一条语句，并在确实是上限触发时指明上限，其余状态保留原有消息：
timeout --kill-after=30 1500 bash -c run_shard || {
rc=$?
elapsed=$(( $(date +%s) - ${E2E_JOB_START_EPOCH:-0} ))
# ... 预算门槛保持不变，但也把状态插值进 ::error:: ...
if (( rc == 124 || rc == 137 )); then
echo "::warning::sandbox:none shard hit its 1500s first-attempt bound on ${RUNNER_NAME:-this runner} after ${elapsed}s; retrying once"
else
echo "::warning::sandbox:none shard failed on ${RUNNER_NAME:-this runner} after ${elapsed}s (attempt exit ${rc}); retrying once (transient shared-host pressure class)"
fi
run_shard
}该步骤运行在 GitHub 的 Linux 默认 shell bash -e {0} 之下，这一点由 scripts/tests/e2e-workflow.test.js 中的 expect(runStep.shell).toBeUndefined() 加上两处缺失的 defaults: 块，以及测试套件的 execFileSync('bash', ['-e', scriptFile]) 共同钉住——所以状态必须如上所示在 || { ... } 组内捕获；裸写 timeout ...; rc=$? 会在 -e 下中止步骤。expect(runStep.run.match(/run_shard/g)).toHaveLength(4) 还意味着新增的捕获不能引入另一个 run_shard 记号，而结尾锚定的 /run_shard\s*\n\s*\}\s*\n\s*fi\s*$/ 要求重试保持为该命令组的最后一条命令。
请扩展 scripts/tests/e2e-shard-retry.test.js 中的 it('retries a first attempt killed by its own wall-clock bound')——它已经驱动 timeoutExit: 124，但只断言了通用的 expect(output).toContain('::warning::')，而普通失败路径同样满足该断言——让它断言上限专属文本，并在 failCalls: '1' 的探针中补上"不包含该文本"的断言；然后移除 rc 捕获并确认该测试变红，再把捕获移到 elapsed= 赋值之后并确认它再次变红、且记录为 exit 0。
— qwen3.8-max via Qwen Code /review (v0.23.0)
| const { exitCode, npmCalls, output, timeoutDurations } = runStepScript({ | ||
| failCalls: '', | ||
| elapsedSeconds: 1200, | ||
| timeoutExit: 124, |
There was a problem hiding this comment.
[Suggestion] R1-5: This is the only execution witness for the new bound, and it asserts a state combination a real exit 124 cannot produce — so it pins the favourable branch only, and the change's headline behaviour has no test capable of failing.
The harness pins the clock: the date stub prints one frozen second and E2E_JOB_START_EPOCH is set to now - elapsedSeconds, and the new timeout stub returns instantly after honouring TIMEOUT_STUB_EXIT, consuming no time at all. So elapsed at the gate is exactly the hand-picked elapsedSeconds and never includes the 1500s the bound actually consumes. On a real runner an exit 124 means the attempt consumed the full bound, so reachable elapsed values are at least setup + 1500 — 1200 is impossible. The two probes that do cover elapsed > 2100 (2101 and 3000) both use failCalls: '1', a shard that reported failure, never timeoutExit: 124, so the branch where a bound-kill lands past the gate and the retry is refused after 1500s of burned pool time has no witness either.
We confirmed the consequence rather than reasoning about it, by driving the step extracted from this commit across an elapsed by exit-124 matrix, and then repeating it on a mutant arm with the cap changed from 1500 to 2400 in a scratch tree. The pinned case stays green across that mutation while the real path inverts — which is exactly the flip a witness needs to be able to make and this one cannot.
To be fair about severity: this stays a Suggestion rather than a Critical because the test is new rather than weakened, and at today's constants the realistic elapsed of setup 179 + bound 1500 = 1679s does still retry. The clock pin itself is legitimate and documented. The cost is that the suite reads as proof of a guarantee it does not test, and the three expect(timeoutDurations).toEqual(['1500']) assertions reinforce that, because they record the requested duration, which bears no relationship to the elapsed clock the gate reads.
Witness:
harness read: dateStub prints a fixed `now`;
E2E_JOB_START_EPOCH = String(now - elapsedSeconds);
timeout stub: if [[ -n "${TIMEOUT_STUB_EXIT:-}" ]]; then exit "$TIMEOUT_STUB_EXIT"; fi
-> consumes no time, so gate elapsed == elapsedSeconds exactly
drives of the step extracted at HEAD:
A elapsed=1200 exit=124 (the pinned case) -> exitCode=0 npmCalls=1 durations=[1500]
::warning:: ... retrying once
B elapsed=1679 exit=124 (real: 179 + 1500) -> exitCode=0 npmCalls=1 durations=[1500]
::warning:: ... retrying once
C elapsed=2400 exit=124 (no witness) -> exitCode=1 npmCalls=0 durations=[1500]
::error:: ... after 2400s ...
not enough left for a retry
MUTANT ARM (cap 1500 -> 2400 in a scratch tree, reverted afterwards):
pinned input elapsed=1200 exit=124 -> exitCode=0 npmCalls=1 durations=[2400]
::warning:: ... retrying once <-- STILL GREEN
real input elapsed=2579 exit=124 -> exitCode=1 npmCalls=0 durations=[2400]
::error:: ... not enough left for a retry
Couple the two clocks arithmetically in the harness rather than picking an elapsed by hand — derive the case's elapsed from the bound the stub recorded (for example a named setup constant plus 1500), or have the timeout stub advance a fake-clock file by the duration it enforces and have the date stub read it — and then assert the outcome that actually follows. Please also add the missing branch as its own case, for instance runStepScript({ failCalls: '', elapsedSeconds: 2400, timeoutExit: 124 }) asserting npmCalls 0, ::error:: and a non-zero exit, so the 124-past-the-gate path is pinned.
The clock must stay deterministic, because the pin is deliberate — scripts/tests/e2e-shard-retry.test.js:56-58 reads "The clock is pinned so the budget-gate boundary cases are exact: with the real clock, elapsed only ever grows between this process and the script's date +%s, which would race the 2100s threshold" — so the coupling has to be stub-driven arithmetic rather than real elapsed time, and the existing elapsedSeconds: 2100 / 2101 boundary cases must stay exact.
For the witness: the new case must go red once its elapsed clock reflects the bound, and must not be re-greened by weakening the assertion. The mutation that proves the coupling is real is changing 1500 to 2400 in the workflow and watching the derived elapsedSeconds move with it — we ran exactly that mutant arm above and the current witness stayed green, which is the gap.
中文说明
这是新上限唯一的执行级见证，而它断言的是一个真实退出码 124 不可能产生的状态组合——因此它只钉住了有利的那一分支，这次改动的核心行为没有任何能让它失败的测试。
测试套件把时钟钉死了：date 桩只打印一个冻结的秒值，E2E_JOB_START_EPOCH 被设为 now - elapsedSeconds，而新的 timeout 桩在响应 TIMEOUT_STUB_EXIT 后立即返回，完全不消耗时间。所以门槛处的 elapsed 恰好等于手工选定的 elapsedSeconds，从不包含上限实际消耗的那 1500 秒。在真实运行器上，退出码 124 意味着该次尝试耗尽了整个上限，因此可达的 elapsed 至少是 setup + 1500——1200 是不可能出现的。真正覆盖 elapsed > 2100 的两个探针（2101 和 3000）都使用 failCalls: '1'，即报告了失败的分片，从不使用 timeoutExit: 124，所以"被上限杀掉后落在门槛之外、在烧掉 1500 秒池时间之后重试被拒绝"这一分支同样没有见证。
我们没有停留在推理上，而是确认了后果：用从本提交提取出的步骤脚本，在 elapsed 与 exit-124 的矩阵上驱动；随后在一次变异臂中把上限从 1500 改为 2400（在临时树中，事后已还原）重复同样的驱动。钉住的用例在该变异下依然全绿，而真实路径的行为却反转了——这正是一个见证必须能做到的翻转，而当前这个做不到。
关于严重程度需要公允说明：这一条保持为 Suggestion 而不是 Critical，因为该测试是新增的而非被削弱的，而且在当前常量下，真实的 elapsed（setup 179 + bound 1500 = 1679 秒）确实仍会重试。时钟钉死本身是合理且有文档说明的。代价在于：套件读起来像是对一个它并未测试的保证的证明，而三处 expect(timeoutDurations).toEqual(['1500']) 断言强化了这种错觉，因为它们记录的是请求的时长，与门槛所读的 elapsed 时钟毫无关系。
见证：
套件阅读：dateStub 打印固定的 `now`；
E2E_JOB_START_EPOCH = String(now - elapsedSeconds)；
timeout 桩：if [[ -n "${TIMEOUT_STUB_EXIT:-}" ]]; then exit "$TIMEOUT_STUB_EXIT"; fi
-> 不消耗时间，所以门槛 elapsed 恰好等于 elapsedSeconds
从 HEAD 提取步骤脚本的驱动结果：
A elapsed=1200 exit=124（被钉住的用例） -> exitCode=0 npmCalls=1 durations=[1500]
::warning:: ... retrying once
B elapsed=1679 exit=124（真实：179+1500）-> exitCode=0 npmCalls=1 durations=[1500]
::warning:: ... retrying once
C elapsed=2400 exit=124（无见证） -> exitCode=1 npmCalls=0 durations=[1500]
::error:: ... after 2400s ...
not enough left for a retry
变异臂（在临时树中把上限 1500 -> 2400，事后已还原）：
钉住的输入 elapsed=1200 exit=124 -> exitCode=0 npmCalls=1 durations=[2400]
::warning:: ... retrying once <-- 依然全绿
真实的输入 elapsed=2579 exit=124 -> exitCode=1 npmCalls=0 durations=[2400]
::error:: ... not enough left for a retry
请在套件中用算术把两个时钟耦合起来，而不是手工挑选 elapsed——例如从桩记录到的上限推导用例的 elapsed（一个具名的 setup 常量加 1500），或者让 timeout 桩按它执行的时长推进一个假时钟文件、再让 date 桩读取它——然后断言实际会产生的结果。也请把缺失的分支补成独立用例，例如 runStepScript({ failCalls: '', elapsedSeconds: 2400, timeoutExit: 124 })，断言 npmCalls 为 0、出现 ::error::、退出码非零，从而钉住"124 落在门槛之外"这条路径。
时钟必须保持确定性，因为这个钉死是刻意的——scripts/tests/e2e-shard-retry.test.js:56-58 写道："The clock is pinned so the budget-gate boundary cases are exact: with the real clock, elapsed only ever grows between this process and the script's date +%s, which would race the 2100s threshold"——所以耦合必须是桩驱动的算术，而不是真实经过的时间，并且现有的 elapsedSeconds: 2100 / 2101 边界用例必须保持精确。
关于见证：新用例在其 elapsed 时钟反映上限之后必须变红，并且不能通过削弱断言把它重新变绿。能证明耦合真实的变异是把工作流中的 1500 改成 2400，并观察推导出的 elapsedSeconds 随之移动——我们上面正是运行了这个变异臂，而当前的见证依然全绿，这就是缺口所在。
— qwen3.8-max via Qwen Code /review (v0.23.0)
|
🔀 Base updated: red check(s) [Classify PR] pass on current main — merged current main via update-branch; CI will re-run. 中文说明🔀 已更新 base：红色检查 [Classify PR] 在当前 main 上通过 —— 已通过 update-branch 合入当前 main，CI 将重新运行。 |
|
Qwen Code review timed out. Qwen review timed out after 10800 seconds (of the 180-minute budget). For large PRs, retry with a longer timeout by commenting: |
* ci: benchmark Vitest forks against E2E shards * ci: prepare E2E fork configuration for review
yiliang114
changed the title
github-actions
Bot
removed
the
review/self-reported
|
Closing because the original first-attempt timeout approach is not safe: the 1500-second cap is below observed healthy shard runtimes, so it can kill green tests and still leave insufficient retry budget. The PR should not be repurposed into a different solution. |
3 participants
What this PR does
Runs each Linux sandbox E2E suite in one job with up to three Vitest forks, replacing three static shard jobs while preserving three-way test concurrency. This is the same patch reviewed and exercised in #11290, forward-ported to
mainbecause #11290 was merged into its stale feature-branch base instead.This PR previously proposed a 1500-second wall-clock bound for the first
sandbox:noneattempt. Review showed that healthy shards already take 26–30 minutes on loaded hosts, so that bound could kill green tests and still leave too little time for a retry. That approach has been reverted; it is no longer present in the final diff.Why it's needed
Issue #11268 exposed the cost of running six separate Linux jobs on the shared ECS pool: repeated setup and host-level contention can make one shard degrade until the existing retry no longer fits inside the job budget. Keeping the same test concurrency inside two runners removes four duplicated jobs and their setup work. The original #11290 run completed both Linux sandbox modes with this layout, but its merge never reached
main.Reviewer Test Plan
How to verify
Confirm that each Linux sandbox creates one job, each job runs the complete suite with at most three Vitest forks, and the existing sandbox-specific exclusions and retry behavior remain unchanged. Confirm the final diff matches #11290 rather than the removed first-attempt timeout proposal.
Evidence (Before & After)
Before: two sandbox modes multiplied by three static shards occupied six pool runners and repeated setup six times.
After: the same two sandbox modes occupy two pool runners and retain three-way test concurrency inside Vitest. The same patch completed the full Linux E2E workflow in #11290.
Tested on
Environment (optional)
The focused workflow test passes: 30 tests. Prettier and
git diff --checkpass. The stable patch ID matches #11290, whose build, typecheck, lint, workflow tests, and full Linux E2E run passed. A repository-wide local actionlint run is currently blocked by unrelated YAML-alias diagnostics inrelease.ymlonmain.Risk & Scope
Linked Issues
Addresses #11268
Forward-port of #11290
中文说明
这个 PR 做了什么
每种 Linux sandbox 的 E2E 套件改为在一个 job 中运行，并最多启用三个 Vitest fork；它替代三个静态 shard job，同时保留三路测试并发。这与 #11290 中已经评审和实际执行过的补丁完全相同。由于 #11290 误合入了旧的 feature 分支而没有进入
main，这里将它补到main。本 PR 之前曾提议给首次
sandbox:none尝试设置 1500 秒墙钟上限。评审发现，繁忙宿主机上的健康 shard 已经需要 26–30 分钟，因此该上限会杀掉原本能够通过的测试，并且仍可能不给重试留下足够时间。该方案已经通过普通 revert 提交撤销，最终 diff 中不再包含它。为什么需要
Issue #11268 暴露了在共享 ECS 资源池上运行六个独立 Linux job 的代价：重复 setup 和宿主机级竞争可能让某个 shard 持续变慢，直到现有重试无法再塞进 job 预算。把相同的测试并发放进两个 runner，可以去掉四个重复 job 及其 setup 工作。#11290 已经用这种布局完整跑过两种 Linux sandbox，但它的合并没有进入
main。Reviewer 测试计划
如何验证
确认每种 Linux sandbox 只创建一个 job，每个 job 使用最多三个 Vitest fork 运行完整套件，并且现有的 sandbox 专属排除项和重试行为保持不变。确认最终 diff 与 #11290 一致，而不是已经撤销的首次尝试 timeout 方案。
证据（修改前后）
修改前：两种 sandbox 模式乘以三个静态 shard，占用六个资源池 runner，并重复执行六次 setup。
修改后：相同的两种 sandbox 模式只占用两个资源池 runner，同时在 Vitest 内保留三路测试并发。相同补丁已经在 #11290 中完成完整 Linux E2E workflow。
测试平台
环境（可选）
定向 workflow 测试 30 个通过。Prettier 与
git diff --check通过。stable patch ID 与 #11290 一致，而 #11290 已通过 build、typecheck、lint、workflow 测试和完整 Linux E2E。本地全仓 actionlint 当前被main上无关的release.ymlYAML alias 诊断阻塞。风险与范围
关联 Issue
处理 #11268
#11290 的 forward-port