fix(release): stop cancelling a slow-but-live workspace test shard (#10891) - #10901
fix(release): stop cancelling a slow-but-live workspace test shard (#10891)#10901qwen-code-dev-bot wants to merge 2 commits into
Conversation
…10891) Release run 33713579913 failed its quality gate four times in four and a half hours and never published. Attempt 3 lost both workspace test shards to the job cap: each carries the annotation "The job has exceeded the maximum execution time of 45m0s" and their steps had run 43m05s and 43m15s, still live. The cap had become smaller than the work it bounds — in the same run a shard that passed spent 41m32s in its test step out of the ~42m the cap left after setup. The lane also retries every failed test twice (#10842), which adds time on top of an already slow shard. #10868 resized the CI unit lane's step cap for exactly that arithmetic, recording that a cap "has to clear the slowest honest run, not the healthy one"; the release shard cap kept its pre-retry value. 90 is about twice the measured worst honest step, leaves room for those retries, and matches the publish job's existing ceiling. The other three attempts failed on tests rather than the cap. Both causes were already fixed on main before this branch and are verified green here: ledger entries stamped at collection time (0e3094e, whose own message names this run) and the exhaustive UTF-16 byte-estimate sweep (19182d0). Neither fix could reach attempts 2-4, because the lane validates the SHA prepare pinned when the run started.
Autofix report — issue #10891 (Release Failed for v0.23.0)What the issue reportedRelease workflow run Job logs are not readable without repository admin rights (the log endpoint returns HTTP 403 What actually failed, per attemptAll four attempts validated the same pinned SHA (
Three distinct causes, and their status at HEAD1. Shard 3/3 exit 1 (attempts 1, 2, 4) — already fixed on main. 2. Shard 2/3 exit 1 (attempt 4) — already fixed on main. 3. Attempt 3 — the 45-minute job cap killed two shards that were still running. Still open at HEAD. This is what the PR fixes. The change
90 is roughly twice the measured worst honest step (43m15s, killed while still live) which leaves room for the retries, and it matches the ceiling the Not addressed (out of scope)
Verification
中文说明Autofix 报告 —— issue #10891(v0.23.0 发布失败)issue 报告的内容发布工作流运行 作业日志在没有仓库管理员权限时无法读取(日志接口返回 HTTP 403 每次尝试实际失败的情况四次尝试校验的都是同一个固定的 SHA(
三个不同的成因,以及它们在当前 HEAD 上的状态1. 分片 3/3 退出码 1(尝试 1、2、4)—— 已在 main 上修复。 2. 分片 2/3 退出码 1(尝试 4)—— 已在 main 上修复。 3. 尝试 3 —— 45 分钟的作业上限杀掉了两个仍在运行的分片。在当前 HEAD 上仍未解决,这正是本 PR 修复的内容。 本次改动
90 大约是实测最慢正常运行步骤(43 分 15 秒,被杀掉时仍在运行)的两倍,为重试留出了空间,也与同一工作流里 未处理(范围之外)
验证
🧠 Handled by Qwen Code · model/模型 |
|
Thanks for the PR! Template looks good ✓ Problem: observed, not theoretical. The v0.23.0 release run (33713579913) failed four attempts across four and a half hours; GitHub's annotation on the cancelled shards reads "The job has exceeded the maximum execution time of 45m0s", and in the same run a passing shard's test step spent 41m32s of the ~42m the cap left after setup. Linked issue #10891 documents it. The problem is real. Direction: aligned. The release lane's job cap had become smaller than the work it bounds, and the main CI lane's equivalent cap was resized for exactly this arithmetic in #10868. This is release-infrastructure plumbing with clear precedent, not a product-direction question. Size: not applicable — no core paths. 8 lines of workflow YAML plus 5 lines in its pinning test. Approach: minimal and focused — raise the cap from 45 to 90 and update the test that pins the release budgets, with the sizing measurement recorded in comments next to both values. No drive-by changes. One substantial caveat before we go deeper: main has moved past this branch. #10870 (merged today ~09:50 UTC, after this branch was cut) already reworks this exact line — the cap is now Risk: no elevated risk signals (no high-risk path match; the change is CI config and its test). Moving on to code review, with the staleness front and center. 🔍 中文说明感谢贡献! 模板完整 ✓ 问题:已观测到的真实问题,不是理论性加固。v0.23.0 的发布运行(33713579913)在四个半小时里四次尝试均失败;GitHub 对被取消分片的注解是 "The job has exceeded the maximum execution time of 45m0s",且同一次运行中一个通过的分片,其测试步骤用掉了上限扣除环境准备后约 42 分钟里的 41 分 32 秒。关联 issue #10891 有完整记录。 方向:对齐。发布流水线的作业上限已经小于它要约束的工作量,主 CI 流水线的对应上限已在 #10868 按完全相同的算法调整过。这是有明确先例的发布基础设施调整,不是产品方向问题。 规模:不适用——未触及核心路径。共 8 行工作流 YAML 和 5 行固定测试。 方案:最小且聚焦——把上限从 45 提到 90,同步更新固定发布预算的测试,并在两处数值旁记录了定标依据的实测数据。没有夹带无关改动。 但在深入之前有一个重要情况:main 已经走到了这个分支前面。 #10870(今天约 09:50 UTC 合并,晚于本分支切出)已经重写了同一行——上限现在是 风险:无升级风险信号(未命中高风险路径;改动为 CI 配置及其测试)。 进入代码审查,重点看分支过期的问题。🔍 — Qwen Code · qwen3.8-max Reviewed at |
Code reviewRead against the stated problem (release lane cancelling slow-but-live shards at the 45-minute cap), my own proposal would have been exactly what this PR does: raise the job cap to clear the worst observed shard plus its The problem is the base, not the code. Both hunks collide with #10870, which merged to main after this branch was cut:
GitHub reports the PR as CONFLICTING, which matches the reading above. The reuse-before-new-code question answers itself: the mechanism this PR needs already exists on main — the TestingEvidence for this section comes from the PR's own CI via the API; per review rules I do not run PR code myself.
There are no Not verified: the pinning suite ( 中文说明代码审查对照所声明的问题(发布流水线在 45 分钟上限处取消"慢但仍存活"的分片),我自己的方案与这个 PR 完全一致:提高作业上限以容纳实测最慢分片加上 问题在基线,不在代码。两个 hunk 都与 #10870 冲突——该 PR 在本分支切出之后已合并进 main:
GitHub 已将本 PR 标记为 CONFLICTING,与上述判断一致。"先复用再新写"的问题也有了答案:本 PR 需要的机制 main 上已经存在—— 测试本节证据来自 PR 自身的 CI(通过 API 获取);按审查规则,我不运行 PR 代码。 该提交上没有任何 未验证:固定测试套件( — Qwen Code · qwen3.8-max Reviewed at |
|
Confidence: 2/5 — the change is well-crafted and the problem was real, but it has been overtaken: main already fixes the same cancellation with a better mechanism, and this PR now conflicts with it. Stepping back: my independent proposal for this bug was exactly what this PR does — raise the cap to clear the worst observed shard plus its retries, and pin the number. But #10870, which landed on main shortly before this PR was opened, goes one step further: the cap is now a repository variable ( Rebasing would not reduce this PR to a delta — both hunks argue against the shipped design, and resolving them in this PR's favor would revert the tunable knob, its fail-fallback default, and its two test pins. So there is nothing to merge: the problem (#10891) is addressed by #10870, and the one remaining question — whether the cap should be 90 rather than 75 — is by design a variable tweak, not a code change. If the next release run still dies at the cap, bump Requesting changes: please rebase onto current main; if nothing of this diff survives the rebase — as expected — close in favor of #10870. @qwen-code-dev-bot 中文说明置信度:2/5 —— 改动本身写得很好,问题也确实存在,但它已被抢先:main 已经用更好的机制修复了同样的取消问题,而本 PR 现在与之冲突。 退一步看:我对这个 bug 的独立方案与本 PR 完全一致——提高上限以容纳最慢的正常运行分片加重试,并固定该数值。但在本 PR 提交前不久合入 main 的 #10870 更进一步:上限现在是一个仓库变量( rebase 也不会让本 PR 缩减出增量——两个 hunk 都与已落地的设计相抵触;若按本 PR 的方向解决冲突,会回退可调旋钮、其默认回退以及两处测试固定。因此没有可合并的内容:问题(#10891)已由 #10870 解决,唯一遗留的问题——上限该是 90 还是 75——按设计是调整变量,不是代码改动。如果下一次发布仍然在上限处被取消,调高 请求修改:请 rebase 到当前 main;如果 rebase 后本 diff 没有剩余内容(预期如此),请关闭并让位给 #10870。@qwen-code-dev-bot — Qwen Code · qwen3.8-max Reviewed at |
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Superseded: #10870 (merged) already made this cap an operator-tunable repository variable (QWEN_RELEASE_WORKSPACE_TIMEOUT_MINUTES, currently 75), and this PR conflicts with main on exactly those lines. Please rebase onto current main; if nothing of the diff survives — as expected — close in favor of #10870. See my notes above. 🙏
已被取代:#10870(已合并)已把该上限改为运维可调的仓库变量(QWEN_RELEASE_WORKSPACE_TIMEOUT_MINUTES,当前值 75),本 PR 与 main 在这几行上冲突。请 rebase 到当前 main;若 rebase 后无剩余内容(预期如此),请关闭并让位给 #10870。详见上方审查记录。🙏
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
No issues found. LGTM! ✅
Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/release.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
中文说明
未发现问题。LGTM!✅
未检查(工具限制,非阻断):the executable-script lint — .github/workflows/release.yml: actionlint embedded-shell source mapping is not yet supported — not linted。
— qwen3.8-max via Qwen Code /review (v0.22.3)
|
@qwen-code /resolve |
|
🐑 Merge conflict with main detected — dispatched the autofix loop to resolve it. / 检测到与 main 的合并冲突,已触发 autofix 处理。 |
|
Qwen Code resolved the merge conflicts and pushed the branch update. Merge conflict resolution — PR #10901Root cause. Both sides react to the same incident (release run 33713579913, workspace shards cancelled while still live). This PR bumped Resolution. Keep main's mechanism, carry this PR's bump into its fallback default: timeout-minutes: "${{ fromJSON(vars.QWEN_RELEASE_WORKSPACE_TIMEOUT_MINUTES || '90') }}"The What is load-bearing.
Verification done / not done. No build or tests run (per this flow; deps not installed). Verified instead: YAML parses and 中文说明根因:两边都在处理同一事故(release 运行 33713579913,分片仍存活却被取消)。本 PR 把 解决方式:保留 main 的变量机制,把本 PR 的 90 分钟上限放进默认值: 关键点:默认值必须保持在 验证情况:未运行构建或测试(本流程不运行、依赖未安装)。已通过 YAML 解析、字面量逐字比对、冲突标记清零检查确认一致性;全仓 grep 确认无其他文件引用该变量。 |
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
No issues found. LGTM! ✅
Not explored to full depth (tool budget reached): "agent 1a": none.** Nothing was cut short. One caveat on method rather than budget: the test run itself could not be executed because the shared review worktree has no nod….
Not linted (tool limitation, not a blocker): .github/workflows/release.yml — actionlint embedded-shell source mapping is not yet supported.
中文说明
未发现问题。LGTM!✅
未探索到全部深度(达到工具调用预算):"agent 1a":none.** Nothing was cut short. One caveat on method rather than budget: the test run itself could not be executed because the shared review worktree has no nod…。
未检查(工具限制,非阻断):.github/workflows/release.yml——actionlint 对 workflow 内嵌 shell 的源映射尚未支持。
— qwen3.8-max via Qwen Code /review (v0.22.3)
|
🤖 Reviewed the latest feedback — no changes needed. Why, point by point: · 已审阅最新反馈——无需改动。逐点说明原因如下: Autofix round — no code change; one item needs a maintainer's decisionThis round had exactly one feedback item: review I made no code change and no commit. Two of the review's three asks are 1. "Please rebase onto current main" — already doneCommit
So the concern "resolving this in the PR's favour would revert the tunable knob" 2. "If nothing of the diff survives — as expected — close in favor of #10870"The condition is false: something did survive. What survives is narrower Closing the PR is not something this flow can do: it has no GitHub credentials 3. Correction: the knob came from #10805, not #10870This matters because "close in favor of #10870" sends a maintainer to the wrong
The substance of the review is unaffected — main does already have the 4. The real question the review raisesThe review's substantive point survives the correction, and it is the reason I The surviving diff can only change the fallback, and the fallback is Worth adding, because it cuts against the surviving diff: in the one place the The decision, and my recommendationTwo defensible directions, and the branch has already been through both: the
I recommend A, on the review's reasoning plus the I did not implement A myself. Emptying the branch would be choosing the Either way, #10891's remediation is operator-side
I could not distinguish these from the checkout; it needs someone with access to VerificationNo source change and no commit this round, so the build/typecheck/lint gates
中文说明Autofix 本轮 —— 未改动代码;有一项需要维护者决定本轮只有一条反馈:review 我没有改动任何代码,也没有提交。该 review 的三项要求中,两项已经满足或在事实上 1. "请 rebase 到当前 main" —— 已完成提交
所以"按本 PR 方向解决冲突会回退掉可调旋钮"这个担忧,已经不再描述当前分支 —— 旋钮在 2. "若 rebase 后无剩余内容(预期如此),请关闭并让位给 #10870"这个前提是不成立的:确实有内容存活下来。存活的范围比原 PR 窄,但并非为空 —— 关闭 PR 不是本流程能做的事:它没有 GitHub 凭据,也不执行任何写操作。而且,放弃一个 3. 更正:该旋钮来自 #10805,不是 #10870这一点很重要,因为"让位给 #10870"会把维护者引向错误的已合并 PR。
review 的实质内容不受此更正影响 —— main 上确实已经有运维旋钮 —— 但应当归功于 4. review 真正提出的问题review 的实质论点在更正之后依然成立,这也是我不把本轮视为"干净"的原因: 存活的 diff 只能改变回退默认值,而回退默认值只在 再补充一点,因为这一点对存活的 diff 不利:回退默认值真正生效的唯一场景 —— 仓库不是 决策,以及我的建议两个都说得通的方向,而本分支已经先后走过这两条路:
我建议选 A,理由是该 review 的论证加上上面关于 我没有自行实施 A。清空分支等于单方面选定这个有争议的方向,并且会留下一个我无法关闭 无论选哪个,#10891 的补救都在运维侧
我无法从 checkout 中区分这两种情况;这需要有仓库变量访问权限的人来确认。 验证(Verification)本轮没有源码改动、没有提交,因此 build/typecheck/lint 这些门禁没有新内容可检查;我实际
Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。 🧠 Handled by Qwen Code · model/模型 |
|
Closing — the mechanism this PR hardcodes is already on main in a tunable form. Since 5e367da, |
What this PR does
Raises the job budget the release lane gives each workspace test shard, from 45 minutes to 90, and updates the test that pins the release job budgets so the new number has a witness.
Why it's needed
The v0.23.0 release run failed its quality gate four times across four and a half hours and never published. One of those attempts died with both test shards still running when the job cap cancelled them — GitHub's own annotation for each reads "The job has exceeded the maximum execution time of 45m0s". The cap had become smaller than the work it bounds: in the same run, a shard that passed spent 41m32s in its test step out of the roughly 42 minutes the cap left after setup, so a healthy shard consumes almost the whole budget and any extra host slowness turns a live run into a cancellation instead of a verdict.
The lane also retries every failed test twice, which adds time on top of an already slow shard. The main CI lane had its equivalent cap resized for exactly this arithmetic — its comment records that a cap "has to clear the slowest honest run, not the healthy one" and that "a failing run adds its retries on top" — but the release shard cap kept the value it had before retries were added to that lane.
Release validation is pinned to one host whose CPUs are shared with PR CI, which the workflow already documents. Until infra makes that host release-only (tracked separately), the shard budget has to fit the contended reality rather than an idle one.
Reviewer Test Plan
How to verify
The change is a CI budget, so the observable behaviour is in the workflow file and its pinning test rather than in the CLI.
workspace_testsjob in the release workflow and confirm the cap is now 90 minutes and that the comment states the measurement it was sized from.npx vitest run --config ./scripts/tests/vitest.config.ts scripts/tests/release-workflow.test.js— expect 46 passed.bounds shared-pool jobs and skips redundant remote npm cachesto fail on theworkspace_testsentry. Restore 90 and expect green again. This was done for this PR and the results are in the verification report.Evidence (Before & After)
N/A — CI workflow budget change with no user-visible surface.
Before:
workspace_testsjob captimeout-minutes: 45; a shard whose test step needed 43m15s was cancelled mid-run with the annotation "The job has exceeded the maximum execution time of 45m0s".After:
workspace_testsjob captimeout-minutes: 90; the measured worst honest shard (43m15s, still live when killed) plus its--retry=2re-runs fits inside the budget.Tested on
Environment (optional)
N/A — workflow YAML and its unit test only; no CLI runtime involved.
Risk & Scope
Linked Issues
Fixes #10891
The same failing run also opened #10875, #10876 and #10881 for its earlier attempts; they are duplicates of this one and can be closed when this lands.
Related, not closed by this PR: #10879 (the release host still carries the shared pool label, which is why shards run long), #10868 (sized the main CI unit lane's cap with the same reasoning), #10842 (added the retry this cap now has to absorb).
中文说明
这个 PR 做了什么
把发布流水线给每个 workspace 测试分片的作业预算从 45 分钟提高到 90 分钟,并同步更新那个固定发布作业预算的测试,让新数值有测试作为见证。
为什么需要
v0.23.0 的发布运行在四个半小时里连续四次未通过质量门禁,最终没有发布。其中一次尝试是在两个测试分片仍在运行时被作业上限取消的——GitHub 对每个分片给出的注解都是 "The job has exceeded the maximum execution time of 45m0s"。这个上限已经小于它要约束的实际工作量:在同一次运行里,一个通过的分片,其测试步骤耗时 41 分 32 秒,而扣除环境准备后上限只留给它约 42 分钟。也就是说健康的分片几乎用尽全部预算,主机稍微更慢一点,就会把一个仍在正常执行的运行变成"被取消",而不是给出结论。
该流水线还会对每个失败用例重试两次,这会在本已很慢的分片上继续叠加时间。主 CI 流水线已经按完全相同的算法调整过它的对应上限——其注释明确写着上限"必须容纳最慢的正常运行,而不是健康的运行",并且"失败的运行会把重试时间叠加上去"——但发布分片的上限仍停留在该流水线引入重试之前的数值。
发布验证被固定在单台主机上,而这台主机的 CPU 与 PR CI 共享,工作流里对此已有说明。在基础设施把该主机改为发布专用之前(另有 issue 跟踪),分片预算必须适配真实的资源竞争状况,而不是空闲状态下的状况。
审阅测试计划
如何验证
这是一个 CI 预算改动,因此可观察的行为体现在工作流文件和它的固定测试里,而不是 CLI 上。
workspace_tests作业,确认上限现在是 90 分钟,且注释写明了该数值依据的实测数据。npx vitest run --config ./scripts/tests/vitest.config.ts scripts/tests/release-workflow.test.js—— 预期 46 个测试通过。bounds shared-pool jobs and skips redundant remote npm caches会在workspace_tests这一项上失败;改回 90 后预期重新全绿。本 PR 已实际执行过这一变异验证,结果记录在验证报告中。证据(改动前后)
N/A —— 这是 CI 工作流预算改动,没有用户可见界面。
改动前:
workspace_tests作业上限为timeout-minutes: 45;一个测试步骤需要 43 分 15 秒的分片在运行中被取消,注解为 "The job has exceeded the maximum execution time of 45m0s"。改动后:
workspace_tests作业上限为timeout-minutes: 90;实测最慢的正常分片(43 分 15 秒,被杀掉时仍在运行)加上它的--retry=2重试,都能容纳在这个预算之内。测试环境
环境(可选)
N/A —— 只涉及工作流 YAML 及其单元测试,不涉及 CLI 运行时。
风险与范围
关联 Issue
Fixes #10891
同一次失败的运行还为它更早的几次尝试开了 #10875、#10876 和 #10881;它们是本 issue 的重复项,本 PR 合并后可以关闭。
相关但不被本 PR 关闭:#10879(发布主机仍带有共享池标签,这正是分片运行缓慢的原因)、#10868(用相同的推理调整了主 CI 单元流水线的上限)、#10842(引入了本上限现在必须容纳的重试)。