ci(ecs): wait out npm publish propagation before updating the runner fleet - #10442
Merged
Merged
Conversation
…fleet `npm publish --provenance` returns before the published version is resolvable — npm prints "Your package is being processed and may take a few minutes to become available" — and release.yml dispatches `npm-published` as soon as it returns. For v0.22.3 the package was published at 17:14Z and only became resolvable at 17:30Z, so the single un-retried `npm view` in "Resolve version" 404'd on ecs-update-sg, ecs-update-64c and ecs-update-hk-1. Only ecs-update-hk-2 succeeded, and only because its job queued for ~3 hours and started after the registry had caught up. That left the fleet split across two CLI versions with no signal: the review and triage workflows install qwen only when it is missing, so the three stale pools kept running 0.22.2 against PRs for a full day while hk-2 ran 0.22.3. Resolve the version once on a hosted runner, polling for up to 25 minutes, and feed the result to the matrix through `needs`. That keeps the registry wait off the ECS pools — which queue behind real review and triage work — and makes every pool install the same version even when their jobs start hours apart.
Collaborator
Collaborator
|
Re-run requested by the author after the Template: the headings deviate from the template ( Problem: an observed incident, not theory. Verified against the API on the first pass: Direction: aligned — this is the repository's own release infrastructure, and the failure mode is silent and long-lived. No CHANGELOG signal applies to CI infrastructure; the incident itself is the direction signal. Size: N/A — no core paths; the change is one workflow plus its test. Approach: the scope is right — poll Risk: no high-risk path matches; no elevated risk signals. Gate passes; moving on. 🔍 中文说明作者在 模板：标题与模板不一致（ 问题：已观测到的事故，不是理论问题。首轮已通过 API 核实：v0.22.3 发布时 方向：对齐——这是仓库自身的发布基础设施，且故障模式静默而长期。CHANGELOG 信号不适用于 CI 基础设施；事故本身就是方向信号。 规模：不适用——未触及核心路径；改动是一个 workflow 及其测试。 方案：范围正确——带截止时间轮询 风险：未命中高风险路径，无升级风险信号。 门通过，继续。🔍 — Qwen Code · qwen3.8-max Reviewed at |
Collaborator
|
Re-run: adjudicating the five outstanding FindingsCritical — the new bash-driven tests are not in the win32 exclude list (confirmed). The chain, verified end to end:
An unbounded Third copy of the workflow-parsing helpers (holds). The resolve step's None of this changes the first-pass read of the production change itself: the resolve-once-on-a-hosted-runner restructure remains correct and minimal, and the harness that executes the real step text is the right shape. The blocker is purely the suite's integration with the gated Windows lane; the four suggestions are cheap pinning fixes in the same "guard the wiring" discipline this PR's own tests practice, and they compose naturally with the Critical fix. Testing — this PR's own CI (unattended run; no PR code executed)Everything that ran on the reviewed commit is green; no failing checks, so no log excerpt to quote. The lane that would have caught the Critical — CI results for
One row per check name (latest run); skipped checks omitted; failures sort first. / 每个检查名一行（取最新一次运行），省略 skipped，失败项排在最前。 What this evidence does and does not cover: the suite pins the resolve loop's logic (transient 404s, budget exhaustion, empty-input 中文说明发现重跑任务：对照受审提交的代码，裁定 Critical——新的 bash 驱动测试不在 win32 排除列表里（确认）。 完整链条已逐环核实：
无上界的 workflow 解析助手的第三份拷贝（成立）。 resolve 步骤的 这些都不改变首轮对生产改动本身的判断：托管 runner 上一次性解析的重构依然正确且最小化，执行真实 step 文本的测试形态也对。阻断项仅是套件与受门控 Windows lane 的集成问题；四条建议与本 PR 自身测试践行的「守卫接线」是同一纪律，都是廉价的钉住补强，可以与 Critical 修复顺手一起完成。 测试——本 PR 自身 CI（无人值守运行；未执行 PR 代码）受审提交上跑过的检查全绿；无失败检查，故无日志摘录可引。本可以抓住 Critical 的 lane—— 证据的边界：套件在 Linux 上钉住了 resolve 循环的逻辑（瞬时 404、预算耗尽、空输入回退 — Qwen Code · qwen3.8-max Reviewed at |
Collaborator
|
Confidence: 2/5 — the workflow fix itself is exactly right, but the PR as pushed turns the repository's only non-Linux CI signal red after merge, and the repo's own rules call that a blocker. My first-pass verdict on the production change stands: a real incident verified against the API, a minimal fix shaped like my independent proposal, and tests that execute the production step text. What changed since then is not the code — the head is the same commit — but the outstanding The four suggestions are the same "guard the wiring" discipline this PR's own tests practice, each a one-to-three-line fix, and they compose naturally with the Critical fix. Once the exclude entry lands — ideally with the pinning fixes alongside — this is back in approve territory without another design conversation. Verdict: request changes. The 中文说明置信度：2/5 —— workflow 修复本身完全正确，但按当前推送合并后，仓库唯一的非 Linux CI 信号会变红，而仓库自己的规则把这当阻断项。 首轮对生产改动的判断不变：经 API 核实的真实事故、与我独立设想一致的最小修复、执行生产步骤文本的测试。此后变化的不是代码——head 仍是同一提交——而是悬而未决的 四条建议与本 PR 自身测试践行的「守卫接线」是同一纪律，每条都是一到三行的修复，可以与 Critical 修复顺手一起完成。补上排除条目（最好连同钉住补强一起）后，无需再次设计讨论即可回到可批准状态。 结论：要求修改。 — Qwen Code · qwen3.8-max Reviewed at |
Contributor
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
qwen-code-ci-bot
approved these changes
Aug 29, 2026
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
LGTM, looks ready to ship — CI landed green after the review. ✅
yiliang114
approved these changes
Aug 29, 2026
yiliang114
left a comment
yiliang114
left a comment
Collaborator
There was a problem hiding this comment.
Approving. The design matches the incident exactly: resolve once on a hosted runner (30s poll, 25-min budget — comfortably past the ~16-min propagation observed for v0.22.3), then all four pools install the same resolved version via needs.resolve.outputs.version, which also fixes the split-version window when pools start hours apart. Stderr is held back per attempt and replayed only on give-up, so logs stay readable. Tests execute the real step body against a stubbed npm (retry-then-success, budget-exhausted with stderr replay, empty-version→latest dist-tag) and pin both consumers to the job output with a guard against a leftover step reference. No blockers.
qwen-code-ci-bot
previously requested changes
Aug 29, 2026
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
Partially reviewed — gaps disclosed.
Not reviewed: build-and-test — Test (windows-latest, Node 22.x) was skipped in CI and its suite was not run locally on Windows.
Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/update-ecs-runner-qwen.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
中文说明
仅完成部分审查，审查缺口已披露。
未审查：build-and-test — Test (windows-latest, Node 22.x) was skipped in CI and its suite was not run locally on Windows。
未检查（工具限制，非阻断）：the executable-script lint — .github/workflows/update-ecs-runner-qwen.yml: actionlint embedded-shell source mapping is not yet supported — not linted。
— qwen3.8-max via Qwen Code /review (v0.22.3)
| encoding: 'utf8', | ||
| env: { | ||
| ...process.env, | ||
| PATH: `${dir}:${process.env.PATH ?? ''}`, |
Collaborator
There was a problem hiding this comment.
[Critical] R1-1: [fails-closed] [regression] The new bash-driven behavioural tests are not covered by any win32 exclude entry in scripts/tests/vitest.config.ts, so the merge_group/schedule/dispatch-gated test_windows lane ('Test (windows-latest, Node 22.x)') collects the suite via npm run test:ci → test:scripts and fails. The exclude list holds pr-self-report-label.test.js, qwen-*-workflow.test.js and serve-ab-workflow.test.js — the glob is anchored at the qwen- prefix, and this file starts with update-, so nothing matches it. On a Windows runner, PATH: \${dir}:${process.env.PATH ?? ''}`joins with:where PATH is;-separated, so the stub npm directory never becomes a discrete PATH entry; the backslash mkdtempSync paths are also interpolated unquoted into the generated bash stub (cat ${counter}), where bash consumes the backslashes as escapes. The extracted script then invokes the real npm(or fails on missing coreutils — ci.yml itself notes "Coreutils like mkdir are not guaranteed on a Git-Bash-only PATH"),readFileSync(counter)throws ENOENT, and all threerunResolve` tests go red. Because the lane is gated, this PR's check page reports it skipped — the first red lands in the merge queue or the nightly, which ci.yml's own comment designates "the repository's only signal about a host that is not Linux" and says to "treat a red nightly as a blocker, not as noise". Before this PR the file was pure-YAML assertions and passed on Windows.
Witness:
win32 collection probe (repo's own matcher, exact exclude list):
target collected on win32 lane: true
controls — serve-ab excluded? true | qwen-resolve excluded? true
picomatch: all three patterns => false for the target
Linux baseline: Tests 7 passed (7)
Suggested fix — add the file to the win32 exclude list (the production step under test is genuinely single-platform, runs-on: 'ubuntu-latest'). The alternative that keeps the two pure-YAML assertions running on Windows is a capability probe with it.runIf(...) on the three runResolve tests:
// scripts/tests/vitest.config.ts — win32 exclude list
'scripts/tests/serve-ab-workflow.test.js',
'scripts/tests/update-ecs-runner-qwen-workflow.test.js',The exclusion must stay win32-conditional and an explicit entry, not a widened glob: scripts/tests/vitest.config.ts keeps the list under process.platform === 'win32' and its comment states "pure YAML-parse workflow suites still do" run on Windows, which non-qwen-* YAML-parse suites rely on. The witness for the fix is the test_windows lane itself: with the entry removed, npm run test:scripts on Windows runs the three runResolve tests and they fail at readFileSync(counter) ENOENT / the status assertions; no in-repo test pins vitest's exclude list.
中文说明
新的 bash 驱动行为测试没有被 scripts/tests/vitest.config.ts 的任何 win32 exclude 条目覆盖，因此仅在 merge_group/schedule/dispatch 触发的 test_windows lane（'Test (windows-latest, Node 22.x)'）会通过 npm run test:ci → test:scripts 收集到该套件并失败。现有排除列表为 pr-self-report-label.test.js、qwen-*-workflow.test.js 和 serve-ab-workflow.test.js——glob 锚定在 qwen- 前缀，而本文件以 update- 开头，任何模式都匹配不到。在 Windows runner 上，PATH: \${dir}:${process.env.PATH ?? ''}`用:拼接，而 Windows 的 PATH 以;分隔，因此 stubnpm 目录不会成为独立的 PATH 条目；mkdtempSync 产生的反斜杠路径还被不带引号地插值进生成的 bash stub（cat ${counter}），bash 会把反斜杠当转义符吃掉。于是提取出的脚本会调用真实的 npm（或因缺少 coreutils 而失败——ci.yml 自己注明「Coreutils like mkdir are not guaranteed on a Git-Bash-only PATH」），readFileSync(counter)抛 ENOENT，三个runResolve` 测试全部变红。由于该 lane 受触发条件限制，本 PR 的检查页显示它为 skipped——第一次变红将出现在合并队列或 nightly，而 ci.yml 的注释明确说这是「the repository's only signal about a host that is not Linux」，并要求「treat a red nightly as a blocker, not as noise」。本 PR 之前该文件只有纯 YAML 字符串断言，在 Windows 上是通过的。
建议修复：把该文件加入 win32 排除列表（被测的生产步骤本就只在 ubuntu-latest 上运行）；若想保留两个纯 YAML 断言在 Windows 上运行，可改用能力探测 + it.runIf(...) 只门控三个 runResolve 测试。修复必须保持在 process.platform === 'win32' 分支内、用显式条目而非放宽 glob——配置注释写明「pure YAML-parse workflow suites still do」，其他非 qwen-* 的纯 YAML 套件依赖这一点。验收标准：移除该排除条目后，test_windows lane 上三个 runResolve 测试会以 ENOENT / 状态断言失败；仓库内没有测试能钉住 vitest 的排除列表，该 lane 本身就是见证。
— qwen3.8-max via Qwen Code /review (v0.22.3)
Comment on lines
+27
to
+28
| runs-on: 'ubuntu-latest' | ||
| timeout-minutes: 30 |
Collaborator
There was a problem hiding this comment.
[Suggestion] R1-5: A single npm view can itself run ~16 minutes under npm's built-in fetch defaults (fetch-retries=2, fetch-timeout=300000ms, 10s/60s backoff — verified via npm config ls -l; the fresh ubuntu-latest resolve job has no overrides), so two slow attempts overshoot the 30-minute job timeout. During a registry stall — a different failure class from the fast-404 propagation this loop was written for — attempt 1 burns the full retry budget (~16 min) and ends still under the 25-minute deadline, the loop sleeps and retries, and attempt 2 crosses timeout-minutes: 30: Actions cancels the job mid-npm view before the designed failure path ever runs. The oncall then sees "The operation was canceled" with no replayed registry stderr and no ::error:: annotation, and the four update jobs show only "skipped" — indistinguishable from an infra flake, defeating exactly the diagnostics this PR added.
Witness:
stalling-registry probe at npm built-in defaults (npm 10.9.8):
conn 00:29:08 -> conn 00:34:18 -> conn 00:40:18 (delta 310s, 360s)
npm error network timeout at: http://127.0.0.1:.../@qwen-code%2fqwen-code
npm-start 00:29:08Z / npm-end 00:45:18Z / npm-exit=1 (16m10s for one invocation)
loop arithmetic: attempt 2 starts ~1000s, can run to ~1970s = 32.8min > 1800s job timeout
Bound each attempt so the deadline check retains control (GNU timeout is available on ubuntu-latest):
version="$(timeout 90 npm view "${specifier}" version 2>"${err_log}" | tail -n 1)" || trueThe per-attempt bound must keep the worst-case total (deadline + one final attempt) under the job timeout — RESOLVE_TIMEOUT_SECONDS: '1500' sits beneath timeout-minutes: 30 in this file. A runResolve variant whose stub npm sleeps past the per-attempt bound pins the fix: without the bound the synchronous spawnSync hangs and vitest's testTimeout fails the test; with it, the script exits 1 within budget.
中文说明
在 npm 内建的 fetch 默认值下（fetch-retries=2、fetch-timeout=300000ms、10s/60s 退避——已用 npm config ls -l 验证；全新的 ubuntu-latest resolve job 没有任何覆盖），单次 npm view 本身就可能跑约 16 分钟，两次慢速尝试就会越过 30 分钟的 job 超时。当 registry 卡住时（与这个轮询循环要解决的快速 404 传播不同的故障类别），第 1 次尝试耗尽全部重试预算（约 16 分钟）后仍未到 25 分钟的 deadline，循环休眠后重试，第 2 次尝试会越过 timeout-minutes: 30：Actions 会在 npm view 中途取消 job，设计好的失败路径（回放 stderr + ::error:: 注解）永远走不到。值班人员只会看到「The operation was canceled」，没有回放的 registry stderr、没有 ::error:: 注解，四个 update job 只显示「skipped」——与基础设施抖动无法区分，恰好废掉了本 PR 新增的诊断能力。
建议给每次尝试加上界，让 deadline 检查保持控制权（ubuntu-latest 上有 GNU timeout，见上方代码块）。每次尝试的上界必须保证最坏情况总时长（deadline + 最后一次尝试）不超过 job 超时——本文件中 RESOLVE_TIMEOUT_SECONDS: '1500' 位于 timeout-minutes: 30 之下。验收标准：新增一个 runResolve 变体，让其 stub npm 在第一次调用时 sleep 超过每次尝试的上界——没有上界时同步的 spawnSync 会挂起、vitest 的 testTimeout 使测试失败；有了上界，脚本会在预算内以 1 退出。
— qwen3.8-max via Qwen Code /review (v0.22.3)
Comment on lines
+35
to
+38
| function stepBody(name) { | ||
| const body = step(name).match(/run: \|-\n([\s\S]*)$/)?.[1] ?? ''; | ||
| return body.replace(/^ {10}/gm, ''); | ||
| } |
Collaborator
There was a problem hiding this comment.
[Suggestion] R1-6: The added step()/stepBody() workflow-parsing helpers are byte-identical copies of the ones already in scripts/tests/qwen-repo-hygiene-workflow.test.js and scripts/tests/qwen-resolve-workflow.test.js — a third copy of a fragile YAML-scanning regex that silently returns '' when it stops matching — while scripts/tests/workflow-helpers.js is the established shared module (7 importers). Any workflow-format change (quote style, step indentation) or a fix to a parsing quirk must now be made in three places, and the copies can drift into differently-weakened assertions with no error. The silent-empty failure mode is not hypothetical: verification demonstrated it live — renaming the step makes step() return '' and every behavioural test dies with a bare ENOENT on a random temp path.
Witness:
mechanical extraction + comparison of the three source files:
all three regex literals identical: true
update-ecs vs repo-hygiene stepBody identical: true
importers of scripts/tests/workflow-helpers.js: 7 files
Consume the shared module instead (getWorkflowJob/getWorkflowStep), hoisting the run: |- body-dedent into workflow-helpers.js so the repo-hygiene copy can collapse into it too:
import { getWorkflowJob, getWorkflowStep } from './workflow-helpers.js';
const resolveStep = getWorkflowStep(getWorkflowJob(workflow, 'resolve'), 'Resolve version');Reuse is constrained by one existing fact: scripts/tests/workflow-helpers.js hard-codes the marker - name: '${stepName}' (6-space indent, single quotes), so the workflow's step naming/indentation must stay exactly that shape — it currently does ( - name: 'Resolve version').
中文说明
新增的 step()/stepBody() workflow 解析助手与 scripts/tests/qwen-repo-hygiene-workflow.test.js 和 scripts/tests/qwen-resolve-workflow.test.js 中已有的实现逐字节相同——这是同一个脆弱的 YAML 扫描正则的第三份拷贝（它在不再匹配时会静默返回 ''）——而 scripts/tests/workflow-helpers.js 才是既有的共享模块（7 处引用）。今后任何 workflow 格式变化（引号风格、步骤缩进）或解析怪癖的修复都必须改三个地方，这些拷贝还可能在无人察觉的情况下各自漂移成不同强度的断言。这种「静默返回空」的失效模式并非假设：验证阶段实际复现了它——把步骤改名后 step() 返回 ''，所有行为测试都以随机临时路径上的裸 ENOENT 死去。
建议改为消费共享模块（getWorkflowJob/getWorkflowStep），并把 run: |- 主体去缩进的逻辑上提到 workflow-helpers.js，让 repo-hygiene 里的那份拷贝也能收敛进去（见上方代码块）。复用受一个既有事实约束：scripts/tests/workflow-helpers.js 硬编码了标记 - name: '${stepName}'（6 空格缩进、单引号），因此 workflow 的步骤命名/缩进必须保持这一形状——目前确实如此（ - name: 'Resolve version'）。
— qwen3.8-max via Qwen Code /review (v0.22.3)
Comment on lines
+74
to
+75
| GITHUB_OUTPUT: ghOutput, | ||
| INPUT_VERSION: '0.22.3', |
Collaborator
There was a problem hiding this comment.
[Suggestion] R1-7: The resolve step's env: declarations (INPUT_VERSION, RESOLVE_TIMEOUT_SECONDS: '1500', RESOLVE_INTERVAL_SECONDS: '30') are pinned by no test, because this harness re-injects its own values for all three. Deleting or renaming any of them in the workflow leaves the whole suite green while breaking every real dispatch: under set -u, deadline=$(( SECONDS + RESOLVE_TIMEOUT_SECONDS )) aborts with "unbound variable" on every release/repository dispatch, all four update jobs show "skipped", and no fleet update happens. Nothing pins the '1500' value itself either — lowering it below the ~16-minute npm propagation delay documented in the workflow's own comment silently reintroduces the fleet-split regression this PR exists to fix. Same unpinned-wiring class as the step-id gap, different edge: this one breaks the workflow→script env edge.
Witness:
ARM 2 (RESOLVE_TIMEOUT_SECONDS line deleted, suite as-is): Tests 7 passed (7)
real dispatch of extracted step body:
line 9: RESOLVE_TIMEOUT_SECONDS: unbound variable, exit=1
ARM 3b (same deletion + suggested fix):
x resolves once on a hosted runner and feeds every pool
-> expected ... to contain 'RESOLVE_TIMEOUT_SECONDS: '1500''
Assert the env declarations in the wiring test:
expect(workflow).toContain("INPUT_VERSION: '${{ inputs.version || github.event.client_payload.version }}'");
expect(workflow).toContain("RESOLVE_TIMEOUT_SECONDS: '1500'");
expect(workflow).toContain("RESOLVE_INTERVAL_SECONDS: '30'");The assertions must match the workflow literals exactly as declared in the resolve step's env block (.github/workflows/update-ecs-runner-qwen.yml:35,44-45). Removing any of the three env: lines from the workflow must turn the wiring test red — proven in verification (ARM 3b / ARM 4).
中文说明
resolve 步骤的 env: 声明（INPUT_VERSION、RESOLVE_TIMEOUT_SECONDS: '1500'、RESOLVE_INTERVAL_SECONDS: '30'）没有任何测试钉住，因为这个测试助手自己重新注入了这三个值。在 workflow 里删除或改名其中任何一项，整个测试套件仍是绿的，但每一次真实 dispatch 都会挂：在 set -u 下，deadline=$(( SECONDS + RESOLVE_TIMEOUT_SECONDS )) 会以「unbound variable」中止，四个 update job 全部显示「skipped」，fleet 更新彻底不发生，且没有任何 CI 信号。'1500' 这个值本身也没被钉住——把它降到 workflow 注释里记录的约 16 分钟 npm 传播延迟之下，就会静默重新引入本 PR 要修复的 fleet 版本割裂回归。这与 step id 未钉住属于同一类「接线未钉住」问题，只是断的是另一条边：这条断的是 workflow→脚本的 env 边。
建议在接线测试中断言这三个 env 声明（见上方代码块）。断言必须与 resolve 步骤 env 块中声明的 workflow 字面量完全一致（.github/workflows/update-ecs-runner-qwen.yml:35,44-45）。验收标准：从 workflow 中删除三条 env: 行中的任意一条，接线测试必须变红——已在验证中证明（ARM 3b / ARM 4）。
— qwen3.8-max via Qwen Code /review (v0.22.3)
Comment on lines
+124
to
+126
| expect(workflow).toContain( | ||
| " version: '${{ steps.version.outputs.version }}'", | ||
| ); |
Collaborator
There was a problem hiding this comment.
[Suggestion] R1-4: This wiring test asserts the job outputs: declaration, needs: 'resolve', and both needs.resolve.outputs.version consumers — but nothing pins id: 'version' on the resolve step, the one link that makes steps.version.outputs.version non-empty. Renaming the id leaves the whole suite green while at runtime outputs.version expands to the empty string, so every pool's Update qwen step installs @qwen-code/qwen-code@. npm resolves the empty tag as latest (measured), so the install itself succeeds and the empty VERSION then fails 'Verify version' loudly — a red fleet update instead of the pinned release, discovered only when the next release runs, not at PR time. The test's own comment ("a leftover step reference would silently expand to an empty version") shows this empty-expansion class is exactly what it intends to guard.
Witness:
mutant (id: 'version' -> id: 'ver', declaration untouched): Tests 7 passed (7)
control: grep for `id: 'version'` in scripts/tests -> zero hits
Suggested change
| expect(workflow).toContain( | |
| " version: '${{ steps.version.outputs.version }}'", | |
| ); | |
| expect(workflow).toContain( | |
| " version: '${{ steps.version.outputs.version }}'", | |
| ); | |
| expect(step('Resolve version')).toContain("id: 'version'"); |
The pinned id must stay version — it is the referent of the job output version: '${{ steps.version.outputs.version }}' in .github/workflows/update-ecs-runner-qwen.yml. Renaming id: 'version' in the workflow must turn the new assertion red.
中文说明
这个接线测试断言了 job 的 outputs: 声明、needs: 'resolve' 以及两处 needs.resolve.outputs.version 消费者——但没有任何断言钉住 resolve 步骤上的 id: 'version'，而它正是让 steps.version.outputs.version 非空的唯一一环。把该 id 改名后整个套件仍是绿的，但运行时 outputs.version 会展开为空字符串，每个池的 Update qwen 步骤就会去安装 @qwen-code/qwen-code@。实测 npm 会把空 tag 解析为 latest，所以安装本身会成功，随后空的 VERSION 会在 'Verify version' 处大声失败——结果是 fleet 更新变红而不是装上钉住的版本，而且要等到下一次发版才被发现，不是在 PR 阶段。该测试自己的注释（「残留的步骤引用会静默展开成空版本」）表明它想防的正是这一类空展开。
验收标准：在 workflow 中把 id: 'version' 改名，新增断言必须变红。钉住的 id 必须保持 version——它是 .github/workflows/update-ecs-runner-qwen.yml 中 job 输出 version: '${{ steps.version.outputs.version }}' 的引用目标。
— qwen3.8-max via Qwen Code /review (v0.22.3)
Collaborator
Author
|
@qwen-code /triage |
Collaborator
|
Sandboxed verification: ✅ passed — merge-ready (agent verdict) - workflow run Ran the PR in an isolated, token-free container: A/B against the base build, mock-free harness assertions, targeted gates. Advisory evidence for human reviewers — not a review, an approval, or a CI check. Scripted assertions: 80 passed · 0 failed · 80 total Flakiness gate: ✅ 1 changed test file(s) x 5 identical rounds, no divergence 中文 — 判定：✅ 通过 · 可合入（agent 判定）沙箱验证在隔离、无凭证的容器中执行了该 PR 的代码（与 base 构建 A/B 对照、无 mock harness 断言、定向门禁）。仅作为评审证据，不构成评审、批准或 CI 检查。 脚本断言：80 通过 · 0 失败 · 80 总计 抖动门：✅ 1 changed test file(s) x 5 identical rounds, no divergence Verification reportPR 10442 verification —
|
| # | arm | npm stub | input | timeout/interval | exit | GITHUB_OUTPUT | attempts | observation |
|---|---|---|---|---|---|---|---|---|
| 1 | base | 3×404 then 0.22.3 | 0.22.3 |
— | 1 | (empty) | 1 | the production race: ::error:: on stdout, 404 stderr in log |
| 2 | head | 3×404 then 0.22.3 | 0.22.3 |
60/0 | 0 | version=0.22.3 |
4 | 3 retry notices; stderr stays clean |
| 3 | base | immediate | 0.22.3 |
— | 0 | version=0.22.3 |
1 | control: base fine when registry is fast |
| 4 | head | immediate | 0.22.3 |
60/0 | 0 | version=0.22.3 |
1 | no retry noise |
| 5 | base | all-404 | 0.22.3 |
— | 1 | (empty) | 1 | old annotation shape |
| 6 | head | all-404 | 0.22.3 |
0/0 | 1 | (empty) | 1 | last stderr replayed; ::error:: … after 0s. on stdout |
| 7 | head | immediate | '' |
60/0 | 0 | version=0.22.3 |
1 | specifier sent: @qwen-code/qwen-code@latest |
| 8 | base | immediate | '' |
— | 0 | version=0.22.3 |
1 | non-regression control, same specifier |
| 9 | head | immediate | v0.22.3 |
60/0 | 0 | version=0.22.3 |
1 | v stripped from specifier |
| 10 | head | 2×ECONNRESET then 0.22.3 | 0.22.3 |
60/0 | 0 | version=0.22.3 |
3 | non-404 npm failures retried too, stderr suppressed |
| 11 | head | 2×404 then 0.22.3 | 0.22.3 |
10/1 | 0 | version=0.22.3 |
3 | real cadence sleeps: 2017 ms ≥ 2×1 s |
Cell 1 vs 2 is the load-bearing flip: identical input, base loses the race (exit 1, one attempt), head waits it out (exit 0, four attempts). Controls (3/4, 7/8) ran on both arms. Worst-case wall time is 1500 s + one trailing 30 s sleep = 25.5 min, inside the 30-min job timeout. The budget cell is speed-independent: with RESOLVE_TIMEOUT_SECONDS=0 the deadline check SECONDS >= deadline holds by the first failed attempt on any machine (SECONDS never decreases), so exactly one attempt is deterministic, not sampled.
Wiring (24/24)
wiring.mjs (log logs/wiring.log): resolve runs on ubuntu-latest with outputs.version = ${{ steps.version.outputs.version }} and the 1500/30 env knobs; update has needs: 'resolve' and both VERSION: consumers read needs.resolve.outputs.version (raw-text census: 2 occurrences, 0 leftover steps.version…); on: triggers, permissions, workflow name, per-pool concurrency group, matrix runners, fail-fast: false, repository guard, and the Update qwen / Verify version step bodies + non-VERSION env are byte-identical to base. release.yml still dispatches event_type=npm-published with client_payload[version], matching the workflow's repository_dispatch.types and INPUT_VERSION expression.
PR's own Testing section, walked step by step
| PR claim | result |
|---|---|
| suite green at head (7/7) | confirmed: Tests 7 passed (7) (also re-asserted in gates.sh) |
| resolves after transient 404s, writes GITHUB_OUTPUT, 404 noise out of log | confirmed (test green; cells 2/10) |
exit 1 + ::error:: on stdout + stderr replayed once budget spent |
confirmed (cell 6) |
still resolves latest with no input version |
confirmed (cells 7/8) |
wiring guard: both consumers read needs.resolve.outputs.version |
confirmed (wiring census; mutant M5) |
negative control: reverting the workflow to main fails 3 of 4 new tests, latest passes on both |
confirmed exactly: 3 failed | 4 passed, red set = {waits out…, fails with the registry error…, resolves once on a hosted runner…}, and each red is an AssertionError with expected-vs-actual values, not a crash (test-grounds.mjs, witness evidence/02-vacuity-and-mutant-matrix.png) |
Mutation matrix (each mutant = one guard crippled in the otherwise-HEAD workflow, suite re-run; witness evidence/02-vacuity-and-mutant-matrix.png, log logs/test-grounds.log):
| mutant | guard crippled | result | pinned by |
|---|---|---|---|
| M0 control | — | 7/7 green | — |
| M1 coarse | whole workflow → base | 3 red / 4 pass | the three new behavioural tests |
| M2 | 2>"${err_log}" suppression |
KILLED | waits out npm publish propagation… |
| M3 | cat "${err_log}" >&2 replay |
KILLED | fails with the registry error… |
| M4 | deadline check (retry forever) | KILLED | fails with the registry error… |
| M5 | one consumer → steps.version.outputs.version |
KILLED | resolves once on a hosted runner… |
| M6 | ::error:: annotation → stderr |
KILLED | fails with the registry error… |
No survivors; every guard the PR introduces is pinned by a named test, and the positive control (M0) proves the harness can go red.
Gates (10/10, witness evidence/03-gates.png, log logs/gates.log)
bash -non both extracted bodies: OK.- shellcheck 0.11.0
--severity=style(with shebang): head clean, base clean (A/A). Without shebang both arms report only SC2148 — an extraction artifact, identical on base. Under the repo wrapper's--enable=allthe only findings are SC2154 onINPUT_VERSION/RESOLVE_*/GITHUB_OUTPUT— variables supplied by the step'senv:block and the Actions runtime, same shape on base; the repo gate never sees them (its shellcheck targets tracked shell files, and its actionlint runs with-shellcheck=). - actionlint 1.7.12 with the repo's flags, whole head tree (51 workflows incl. the changed one): 0 findings. A base-file probe shows only the pre-existing
ecs-update-hk-*runner-label warnings, which the repo's own ignore list covers. - eslint on the changed test file: clean.
- Liveness controls: planted shellcheck violation caught (exit 1); planted unused var caught by eslint (exit 1), file restored clean (exit 0); prettier-YAML is not a live gate here —
prettier --checkfails on 12/51 workflow files repo-wide, and the changed file fails identically at base on the untouched matrix line, so the drift is pre-existing and not a PR finding. - yamllint: could not run in this container —
lint.js --setupfails (pip3: Permission denied),python3 -m pipdoes not exist (EXTERNALLY-MANAGED), apt requires root (uid 1000). The PR's own CI lint lane covers it.
Findings
None blocking. Non-blocking observations, for the record:
- The retry loop retries any empty-stdout npm failure, not only 404s (cell 10: ECONNRESET retried). This matches the intent ("wait the registry out") and the replayed last stderr preserves the reason; noted, not a defect.
- Only the last attempt's stderr survives to the replay (
2>truncates per attempt) — exactly as the description states. - The PR's follow-up claim is accurate:
main-ci-failure-issue.ymlwatches['E2E Tests', 'SDK Python', 'Qwen Code CI']and its event filter excludes dispatch-triggered runs, so failures ofupdate-ecs-runner-qwenstill file no issue — the declared follow-up is real.
Not covered
- yamllint — infeasible in-container (see Gates); covered by the PR's own CI lint lane.
- Calibration against the real registry — the A/B reproduces the wire shape of the incident (npm-shaped 404 stderr, single-vs-retried
npm view) with a stub, not real npm/registry timing. No GitHub token/Actions API here, so a byte-for-byte replay of run 33193932104's realResolve versionlog (the artifact that would have calibrated the replay) was not retrievable; state of the actual fleet and the out-of-band repair are likewise outside the sandbox. - ECS-pool side —
Update qwen/Verify versionwere not executed against a real runner; both step bodies are byte-identical to base (wiring check), so the risk surface there is unchanged. - Day-one cost of the hosted wait — ≤25 min of
ubuntu-latestper dispatch, versus four ECS pools each doing a seconds-longnpm viewbefore; dispatch frequency is not measurable from a depth-2 clone. A maintainer can confirm withgh api "repos/QwenLM/qwen-code/actions/workflows/update-ecs-runner-qwen.yml/runs?per_page=100" --jq '.workflow_runs | length'over a window. - Repo-wide gates — full
eslint --max-warnings 0and the full test suite were not re-run (targeted runs only; the PR's CI covers them). - Concurrent dispatches —
resolvehas no concurrency group; two overlapping dispatches would each feed their own matrix, serialized per-runner by the unchangedupdateconcurrency groups. Same exposure as base (which had no workflow-level group either), so not a regression.
Methodology
Environment: the CI verify container (node:22-bookworm, bash 5.2.15); the changed step now targets ubuntu-latest (bash 5.2 — same major). Step bodies were extracted verbatim by parsing the YAML with js-yaml (extract-step.mjs), never retyped, and executed via bash --noprofile --norc with a stubbed npm on PATH encoding npm's E404 wire shape; every observable (exit code, GITHUB_OUTPUT, attempt counter, stdout/stderr, exact specifier argv) was asserted by script. Wiring was asserted from the parsed YAML plus raw-text censuses. Vacuity and the mutant matrix ran the PR's own vitest suite (scripts/tests/vitest.config.ts) from a scratch worktree at HEAD^1 with the HEAD test file. Gates used the repo's pinned linters via scripts/lint.js --setup (individual --actionlint/--shellcheck checks only — never the no-arg form, which runs prettier --write .). Raw logs in logs/, harnesses (ab-resolve.mjs, wiring.mjs, test-grounds.mjs, gates.sh) and per-cell evidence in evidence/; scratch worktree removed after capture.
Flakiness gate log
rounds=5 files=1 skipped=0
file scripts/tests/update-ecs-runner-qwen-workflow.test.js: (cd .) npx --no-install vitest run --config ./scripts/tests/vitest.config.ts ./scripts/tests/update-ecs-runner-qwen-workflow.test.js
per-file results (P=pass F=fail I=infra-exit, one letter per run):
scripts/tests/update-ecs-runner-qwen-workflow.test.js: PPPPP
verdict: pass
summary: 1 changed test file(s) x 5 identical rounds, no divergence
--- per-invocation detail (full copy in the artifact) ---
round 1 · scripts/tests/update-ecs-runner-qwen-workflow.test.js: P (exit 0)
round 2 · scripts/tests/update-ecs-runner-qwen-workflow.test.js: P (exit 0)
round 3 · scripts/tests/update-ecs-runner-qwen-workflow.test.js: P (exit 0)
round 4 · scripts/tests/update-ecs-runner-qwen-workflow.test.js: P (exit 0)
round 5 · scripts/tests/update-ecs-runner-qwen-workflow.test.js: P (exit 0)
Evidence images
Harness scripts and raw logs are in the workflow run artifacts (7-day retention).
— Qwen Code · sandboxed verification
Collaborator
|
Triage re-run completed without a new review. The bot already has a review of its own on 机器人在 The stage comments above were updated with the latest result. View workflow run. 上方各阶段评论已更新为最新结果。查看工作流运行。 |
wenshao
enabled auto-merge
August 29, 2026 04:12
qqqys
approved these changes
Aug 29, 2026
github-actions
Bot
added
the
skip-changelog-auto
Collaborator
|
Released in v0.23.0. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Update ECS Runner Qwenis what keeps the system-wideqwenon the ECS runner pools current. On the v0.22.3 release it failed on 3 of its 4 pools, and nothing noticed.Timeline from release run 33188987718 and update run 33193932104:
17:14:16npm publish --provenancereturns+ @qwen-code/qwen-code@0.22.3, together withYour package is being processed and may take a few minutes to become available.17:16:39release.ymldispatchesrepository_dispatch: npm-publishedwith version0.22.317:16:47–17:17:13Resolve versionrunsnpm view @qwen-code/qwen-code@0.22.3 versiononecs-update-sg,ecs-update-64c,ecs-update-hk-1→npm error 404 No match found for version 0.22.3, no retry,exit 117:30:3520:06:17ecs-update-hk-2— queued behind ~3 hours of other work — finally starts, resolves0.22.3and installs itnpm's publish pipeline is asynchronous, so
npm publishreturning is not the same as the version being resolvable; here the gap was ~16 minutes. The single un-retriednpm viewlost that race.The consequence is silent and long-lived.
qwen-code-pr-review.ymlandqwen-triage.ymlinstall the CLI only when it is absent:On a self-hosted runner
/usr/bin/qwenalways exists, so the three stale pools kept reviewing PRs with0.22.2for a full day whilehk-2ran0.22.3— e.g. this review job onecs-qwen-runner-sg-18printed0.22.2more than five hours after the release.Change
Split the workflow into a
resolvejob and the existingupdatematrix:resolveruns onubuntu-latestand pollsnpm viewevery 30s for up to 25 minutes, instead of failing on the first 404. Per-attempt stderr is suppressed so ~50 identical 404 blocks do not bury the log, and the last one is replayed if the wait is exhausted.needs.resolve.outputs.version.Resolving once on a hosted runner rather than per-pool has two benefits beyond the retry: the registry wait does not occupy ECS runners that queue behind real review and triage work, and every pool installs the same version even when their jobs start hours apart — which is exactly the divergence that produced the split fleet here.
Behaviour that is deliberately unchanged: an empty dispatch input still resolves the
latestdist-tag; thenpm install -gretry loop, theNPM_CONFIG_PREFIXhandling and theVerify versioncheck are untouched.Testing
scripts/tests/update-ecs-runner-qwen-workflow.test.jsgains behavioural coverage: theResolve versionstep body is extracted from the YAML and executed underbashagainst a stubbednpmthat 404s a configurable number of times.GITHUB_OUTPUT, keeps the 404 noise out of the log::error::annotation on stdout and the registry's stderr replayed once the budget is spentlatestdist-tag with no input versionVERSION:consumers readneeds.resolve.outputs.version(a leftoversteps.version.outputs.versionwould expand to empty and install@qwen-code/qwen-code@)Negative control: reverting
.github/workflows/update-ecs-runner-qwen.ymltomainfails 3 of the 4 new tests, and the 4th (latestdist-tag) passes on both as a non-regression guard.The fleet itself was repaired out-of-band by a manual
workflow_dispatchof the workflow;ecs-update-sgis back on0.22.3and the remaining pools are queued.Follow-up (not in this PR)
This workflow's failures are not covered by
main-ci-failure-issue.yml, so a broken fleet update produces no issue and no notification. Worth wiring up separately.中文说明
问题
Update ECS Runner Qwen负责把 ECS runner 池上系统级的qwen升级到最新版。v0.22.3 发布时，它 4 个池挂了 3 个，而且没有任何人收到通知。时间线（来自 release run 33188987718 与 update run 33193932104）：
17:14:16npm publish --provenance返回+ @qwen-code/qwen-code@0.22.3，同时打印Your package is being processed and may take a few minutes to become available.17:16:39release.yml发出repository_dispatch: npm-published，版本0.22.317:16:47–17:17:13Resolve version在ecs-update-sg、ecs-update-64c、ecs-update-hk-1上执行npm view @qwen-code/qwen-code@0.22.3 version→npm error 404 No match found for version 0.22.3，没有重试，直接exit 117:30:3520:06:17ecs-update-hk-2排队约 3 小时后才启动，此时解析成功并装上了0.22.3npm 的发布流水线是异步的，
npm publish返回 ≠ 版本可解析，这次间隔约 16 分钟。那一次没有重试的npm view正好输在这个竞态上。后果是静默且持续的。
qwen-code-pr-review.yml和qwen-triage.yml都是「不存在才安装」：self-hosted runner 上
/usr/bin/qwen一直在，所以那 3 个没升级的池整整一天都在用0.22.2评审 PR，而hk-2用的是0.22.3——例如这个跑在ecs-qwen-runner-sg-18上的评审 job，在发版 5 个多小时之后打印的仍然是0.22.2。改动
把 workflow 拆成
resolvejob 和原有的update矩阵：resolve跑在ubuntu-latest上，每 30 秒轮询一次npm view，最长等待 25 分钟，而不是第一次 404 就失败。逐次的 stderr 被暂存，避免约 50 个重复的 404 段落淹没日志；等待耗尽时再把最后一次回放出来。needs.resolve.outputs.version。只解析一次（而不是每个池各解析一次）除了重试之外还有两个好处：等待 registry 的时间不再占用要排队跑真实评审/triage 任务的 ECS runner；并且即使各池的 job 相隔数小时才启动，装的也是同一个版本——而这正是本次 fleet 版本割裂的直接成因。
刻意保持不变的行为：dispatch 时不填版本仍然解析
latestdist-tag；npm install -g的重试循环、NPM_CONFIG_PREFIX处理和Verify version校验都没有动。测试
scripts/tests/update-ecs-runner-qwen-workflow.test.js增加了行为级覆盖：从 YAML 中提取Resolve version的 step body，在bash下针对一个可配置 404 次数的npm桩执行。GITHUB_OUTPUT，并且 404 噪音不进日志::error::注解落在 stdout，registry 的 stderr 被回放一次latestdist-tagVERSION:都读needs.resolve.outputs.version（残留的steps.version.outputs.version会展开成空串，导致去装@qwen-code/qwen-code@）反向对照：把
.github/workflows/update-ecs-runner-qwen.yml还原成main的版本后，4 个新用例挂 3 个；第 4 个（latestdist-tag）两边都过，它本来就是防回归守卫。fleet 本身已通过手动
workflow_dispatch带外修复：ecs-update-sg已回到0.22.3，其余池在排队。后续（不在本 PR 内）
这个 workflow 的失败不在
main-ci-failure-issue.yml的覆盖范围内，所以 fleet 更新挂掉既不会开 issue 也不会有通知，值得另开一个 PR 接上。