ci(ecs): file an issue when the runner fleet update fails - #10445
Conversation
…fleet `npm publish --provenance` returns before the published version is resolvable — npm prints "Your package is being processed and may take a few minutes to become available" — and release.yml dispatches `npm-published` as soon as it returns. For v0.22.3 the package was published at 17:14Z and only became resolvable at 17:30Z, so the single un-retried `npm view` in "Resolve version" 404'd on ecs-update-sg, ecs-update-64c and ecs-update-hk-1. Only ecs-update-hk-2 succeeded, and only because its job queued for ~3 hours and started after the registry had caught up. That left the fleet split across two CLI versions with no signal: the review and triage workflows install qwen only when it is missing, so the three stale pools kept running 0.22.2 against PRs for a full day while hk-2 ran 0.22.3. Resolve the version once on a hosted runner, polling for up to 25 minutes, and feed the result to the matrix through `needs`. That keeps the registry wait off the ECS pools — which queue behind real review and triage work — and makes every pool install the same version even when their jobs start hours apart.
A failed fleet update produces no signal today. `main-ci-failure-issue.yml` cannot cover it — that workflow watches test suites on `main`, while this one runs off `repository_dispatch` — so the only trace is a red run in the Actions tab that nobody is looking at. The silence is what makes it expensive. `qwen-code-pr-review.yml` and `qwen-triage.yml` install the CLI only when `command -v qwen` finds nothing, and on a self-hosted runner it never does, so a pool that misses an update keeps answering PRs on the old version indefinitely and looks exactly like a healthy one. On v0.22.3 three of the four pools failed and the split fleet ran for a full day before anyone read a version string. Add a `report_failure` job that files, or comments on, a single issue naming the pools left behind. Details: - `cancelled` is deliberately not treated as a failure: the per-pool concurrency group cancels an older dispatch's pending legs whenever a newer one arrives, which says nothing about the fleet. - The pools are read from this run's job conclusions, since a matrix job's per-leg results are not reachable through `needs`. - Dedup is by an exact body marker matched client-side, scoped by `scope/ci-cd`, following `.github/scripts/image-build-failure-issue.sh`: GitHub search tokenizes such a marker apart, so a search-based lookup never finds what the script files. Both labels are applied at creation so the dedup key can never be half-written. - The job is hosted, not on the ECS pool it reports on.
|
Thanks for the PR!
Moving on to code review. 🔍 中文说明感谢贡献!
进入代码审查。🔍 — Qwen Code · qwen3.8-max Reviewed at |
Code reviewOne blocker: Everything else reads clean:
TestingPR orchestration checks on
No Sandboxed verification would settle this ahead of the retarget: 中文说明代码审查:一个阻塞项—— 其余部分干净: 测试:本提交没有 — Qwen Code · qwen3.8-max Reviewed at |
|
Confidence: 2/5 — the design is right and the tests are the right shape, but as written the new job cannot run: its checkout has no Everything upstream of that one line is genuinely good, and I want to say so plainly: the hosted-runner reasoning, Requesting changes on that single point; with it fixed this is ready to ship. 🙏 中文说明置信度:2/5——设计正确、测试形态也对,但按现状这个新 job 跑不起来:它的 checkout 没有 除这一行之外确实都好,值得直说:托管 runner 的理由、 仅就这一点请求修改;修好后即可合入。🙏 — Qwen Code · qwen3.8-max Reviewed at |
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
One blocker: the new report_failure job runs actions/checkout without contents: read in its job-level permissions, so the checkout fails and the stale-fleet issue is never filed — see my Stage 2 notes for the fix and the repo precedent. 🙏
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Partially reviewed — gaps disclosed.
Not reviewed: reverse audit — stopped before round 5 by the review time budget.
Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/update-ecs-runner-qwen.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
中文说明
仅完成部分审查,审查缺口已披露。
未审查:反向审计——评审时间预算不足,未能开始第 5 轮。
未检查(工具限制,非阻断):the executable-script lint — .github/workflows/update-ecs-runner-qwen.yml: actionlint embedded-shell source mapping is not yet supported — not linted。
— qwen3.8-max via Qwen Code /review (v0.22.3)
CanReader
left a comment
There was a problem hiding this comment.
The gap this closes is real and the "name the pools, not just the run" instinct is the most valuable part of the change.
The invisibility argument checks out as written: because qwen-code-pr-review.yml and qwen-triage.yml install the CLI only when command -v qwen finds nothing, and a self-hosted runner always has it, a pool that misses an update keeps serving PRs on the old version with no external signal. That is a genuinely silent failure — the fleet looks healthy because every job succeeds, just against a stale CLI — and it is not the kind of thing main-ci-failure-issue.yml can cover, since that watches suites on main while this workflow runs off repository_dispatch. Citing the v0.22.3 incident where three of four pools 404'd on an unpublished version, and the split went unnoticed for a day, is the right justification: this is a fix for a known observed outage, not a hypothetical.
Reading per-leg conclusions back out of the jobs API rather than needs is the correct workaround. A matrix job's individual leg outcomes genuinely are not reachable through needs — you get the aggregate — so filtering .jobs[] | select(.name | startswith("Update Qwen on ")) on failure/timed_out is the only way to name the specific pools. Including timed_out alongside failure matters here, since a pool wedged on a slow npm fetch is exactly the case that produced the original incident.
Two things worth confirming:
- The job-name prefix match is a string contract between the script and the matrix's
name:template. If someone renames the matrix jobs, this silently selects nothing and the issue body reports no failed pools while the workflow still files an issue — the failure mode is a misleading issue rather than a missing one. Ifupdate-ecs-runner-qwen-workflow.test.jsdoes not already pin that prefix against the workflow's actual job name template, that is a cheap assertion to add and it is the one thing here most likely to rot. set -euo pipefailwith agh api ... --jqinto a command substitution: if the API call fails (rate limit, transient 5xx),failedends up empty and the script continues under-eonly if the substitution's status is masked by the assignment. Worth confirming the script distinguishes "the API call failed" from "no pools failed", because those two produce very different issues.
Filing-or-updating a single issue rather than one per run is the right shape — a fleet that stays stale for a day should not produce a day's worth of duplicates.
…-issue # Conflicts: # scripts/tests/update-ecs-runner-qwen-workflow.test.js
…dedup lookup R1-1: a job-level `permissions:` block replaces the workflow-level one, so `report_failure` held only `actions: read` + `issues: write` and its `actions/checkout` could not fetch the ref — the job that exists to break the silence would have 403'd on every run. Spell out `contents: 'read'`. R1-2: the bash replays (both the resolve ones this suite already carried and the new report ones) move behind the same `process.platform !== 'win32' && jq` gate the sibling image-build suite uses. The `:`-joined PATH and the backslash `RUNNER_TEMP` cannot be expressed on the Windows lane; the YAML-parse tests stay ungated so they still run there. R1-3/R1-7/R1-8: the dedup lookup both failure reporters carried byte-identical moves into `.github/scripts/find-marked-issue.sh`, so the contract is fixed in one place. It gains the two guards that copy needed: `(.body // "")`, because jq's `contains()` exits 5 on a null body and one bodyless `scope/ci-cd` issue would abort the script before anything is filed; and a 1000-issue ceiling instead of a 200-issue newest-first window, because the stale-fleet issue is opened once and only ever commented on, so it drifts to the oldest slot and would eventually fall out and start filing duplicates. R1-4: the fixture pool names and the job-name prefix are now derived from the workflow itself, and a test pins the prefix in the script against the `update` job's `name:` template. A rename used to pass every test while naming zero pools in production. R1-5/R1-6: the jobs read reports how many pool legs ran alongside which are stale, and its own exit status is captured rather than swallowed. The body now has three honest shapes — pools named; "the run failed before the pool matrix started" when `resolve` failed and no leg ever ran (it no longer points the operator at a `Verify version` step that does not exist); and "this run's job conclusions could not be read" when the jobs API call fails, which under `set -euo pipefail` used to be indistinguishable from "no pool failed". The workflow's recorded size baseline is bumped to its actual 7187 bytes: it was 3303 from before the resolve job landed, and the remaining allowance was down to 212 bytes.
|
All 8 threads from round 1 are addressed and resolved in Both Criticals were real and are fixed. R1-1 — a job-level Three of the six Suggestions turned out to be one finding. R1-3 (the byte-identical dedup copy), R1-7 (a 200-issue newest-first window an immortal issue falls out of) and R1-8 ( R1-4, R1-5 and R1-6 are taken as written. R1-4 also caught a live drift — @CanReader — both of your confirmations were worth asking:
Verification. 45 tests across this suite and the sibling image-build one. Mutation matrix: 9 mutants, 9 killed, each reding exactly the test named for it — drop 中文说明第 1 轮的 8 条线程已全部处理并 resolve( 两条 Critical 都成立,都已修。 R1-1——job 级的 六条 Suggestion 里有三条其实是同一个发现。 R1-3(逐字节重复的去重代码)、R1-7(200 条「最新优先」窗口会漏掉永久 issue)、R1-8( R1-4、R1-5、R1-6 按报告采纳。R1-4 还顺带抓到一处已经发生的漂移—— @CanReader——你的两点确认都问对了:
验证。 本套件 + 兄弟 image-build 套件共 45 个用例通过。变异矩阵:9 个变异体、9 个全杀,每个只打红为它写的那个用例——删 |
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/update-ecs-runner-qwen.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
Deferred under the convergence posture (round 2, not a blocker) — recorded, not requested in this round:
scripts/tests/update-ecs-runner-qwen-workflow.test.js:417 — [probe] comment-path replay pins routing but not body attachment (deferred by the code-age rule — code unchanged since round 1)
[Critical] .github/scripts/ecs-fleet-update-failure-issue.sh:32 SC2016 — Expressions don't expand in single quotes, use double quotes for that. [lint]
[Critical] .github/scripts/ecs-fleet-update-failure-issue.sh:50 SC2016 — Expressions don't expand in single quotes, use double quotes for that. [lint]
[Critical] .github/scripts/ecs-fleet-update-failure-issue.sh:52 SC2016 — Expressions don't expand in single quotes, use double quotes for that. [lint]
[Critical] .github/scripts/ecs-fleet-update-failure-issue.sh:57 SC2016 — Expressions don't expand in single quotes, use double quotes for that. [lint]
[Critical] .github/scripts/ecs-fleet-update-failure-issue.sh:59 SC2016 — Expressions don't expand in single quotes, use double quotes for that. [lint]
[Critical] .github/scripts/ecs-fleet-update-failure-issue.sh:61 SC2016 — Expressions don't expand in single quotes, use double quotes for that. [lint]
[Critical] .github/scripts/ecs-fleet-update-failure-issue.sh:63 SC2016 — Expressions don't expand in single quotes, use double quotes for that. [lint]
中文说明
未检查(工具限制,非阻断):the executable-script lint — .github/workflows/update-ecs-runner-qwen.yml: actionlint embedded-shell source mapping is not yet supported — not linted。
收敛姿态下延后(第 2 轮,非阻断)——已记录,本轮不要求修改:共 1 条(原文未翻译,列表见上方英文部分)。
[Critical] .github/scripts/ecs-fleet-update-failure-issue.sh:32 SC2016 — Expressions don't expand in single quotes, use double quotes for that. [lint]
[Critical] .github/scripts/ecs-fleet-update-failure-issue.sh:50 SC2016 — Expressions don't expand in single quotes, use double quotes for that. [lint]
[Critical] .github/scripts/ecs-fleet-update-failure-issue.sh:52 SC2016 — Expressions don't expand in single quotes, use double quotes for that. [lint]
[Critical] .github/scripts/ecs-fleet-update-failure-issue.sh:57 SC2016 — Expressions don't expand in single quotes, use double quotes for that. [lint]
[Critical] .github/scripts/ecs-fleet-update-failure-issue.sh:59 SC2016 — Expressions don't expand in single quotes, use double quotes for that. [lint]
[Critical] .github/scripts/ecs-fleet-update-failure-issue.sh:61 SC2016 — Expressions don't expand in single quotes, use double quotes for that. [lint]
[Critical] .github/scripts/ecs-fleet-update-failure-issue.sh:63 SC2016 — Expressions don't expand in single quotes, use double quotes for that. [lint]
— qwen3.8-max via Qwen Code /review (v0.22.3)
…erting an unread shape R1-6 (round 1, reopened): the `legs == 0` branch was unreachable. When `resolve` fails the matrix is skipped, but the jobs API still lists every leg under its fully expanded name, so the count was 5 rather than 0 and the body fell through to the pool-level wording it exists to avoid. The filter now drops `skipped` legs, and the resolve-failure fixture carries the five skipped legs a real run produces instead of pinning a shape the API never returns. R2-2: the comment claimed `set -e` does not fire on a command substitution feeding an assignment. It does — `bash -c 'set -e; x=$(false); echo reached'` exits 1 with no output — so `|| jobs_status=$?` is load-bearing, and the comment now says so rather than teaching the next reader to omit it. R2-1: the dedup lookup was the one API call with no guard, so a transient 5xx on it killed the reporter before anything was filed. Both callers now degrade to "no known issue" and file; a duplicate costs less than the silence. R2-3: the jobs-API-unreadable branch asserted a pool-level failure and named `Verify version` steps. Which shape failed is exactly what could not be read — `resolve` may have failed before any pool ran — so it now says that and points at the run, reading `Resolve version` first when the version is unresolved. R2-4: the lookup took the FIRST marker match of a newest-first listing, so an issue that merely quotes the marker outranks the canonical one permanently — worse for the image-build caller, whose recurrence path rewrites the selected issue's body. `last(...)` takes the oldest open match, which is the issue a reporter opened for itself. R2-5..R2-11: the seven SC2016 notes came from moving backtick-bearing strings out of the printf block's `disable`. The directive now sits on the `if` that carries them and on the jq filter, so the rest of the file keeps the check.
|
Round 2 addressed in R1-6 was right to come back. The R2-2 was a genuine error in my comment, not a nit. I wrote that R2-3 and R2-4 taken as written. The jobs-API-unreadable branch no longer asserts which shape failed — that is precisely what could not be read — and the lookup takes the oldest open marker match ( I declined one sub-suggestion: R2-3 offered a retry of the SC2016 ×7 — real, and mine: moving the backtick-bearing strings out of the Verification. 49 tests across this suite and the sibling image-build one. Round-2 mutation matrix: 7 mutants, 7 killed — count skipped legs again; restore the pool-level headline on the unread branch; unguard either lookup; revert The two red jobs on
Neither can be reached from 中文说明第 2 轮已在 R1-6 回归得有道理。 我在第 1 轮加的 R2-2 是我注释里的真错误,不是吹毛求疵。 我写的是「 R2-3 与 R2-4 按报告采纳。 jobs API 读不到的分支不再断言是哪种形态失败——那恰恰是读不到的东西;查找改为取最旧的 marker 匹配( 有一条子建议我没有采纳:R2-3 提出对 SC2016 ×7——成立,而且是我引入的:把带反引号的字符串移出 验证。 本套件 + 兄弟 image-build 套件共 49 个用例通过。第 2 轮变异矩阵:7 个变异体、7 个全杀——重新把 skipped leg 计入;把池级首句放回「读不到」分支;分别去掉两个查找的守卫;把
这两个都不可能由 |
CI runs `node scripts/lint.js --prettier` — which is `prettier --write .` — before the suite, and prettier reflows the workflow's inline `runner: [...]` array onto one line per entry. The parser only matched the checked-in layout, so on CI it found zero pools: `filters the run jobs by the prefix the matrix job actually uses` failed on `pools.length`, and `falls back when the legs ran but none reported a failure` built a fixture with no legs and took the resolve-only body shape instead. This is the first CI run that reached the suite at all. `test:ci` is `npm run test:ci --workspaces && npm run test:scripts`, and on the previous head a web-shell unit test failed in the workspace half, short-circuiting the scripts half before it ran. `parsePools()` now accepts both layouts and the prefix-contract test pins both against literal fixtures, so neither reformatting nor un-reformatting the workflow can take the suite red again. Verified by running the suite twice — once on the checked-in file, once after `prettier --write` on it — 21 passed both ways.
|
Correction and follow-up on the CI note above: the run on CI runs runner: ['ecs-update-hk-1', 'ecs-update-hk-2', …] # checked in
runner: # what the suite sees on CI
[
'ecs-update-hk-1',
…
]My pool parser matched only the checked-in layout, so on CI it found zero pools. Two tests fell over: Why the previous run did not catch it:
Confirmed on CI. The next run went red once more on That is the first CI execution of either suite on this PR, and it passes with the workflow in whatever layout 中文说明对上面那条 CI 说明的更正与后续: CI 在跑套件之前会执行 runner: ['ecs-update-hk-1', 'ecs-update-hk-2', …] # 仓库里的样子
runner: # CI 上套件看到的样子
[
'ecs-update-hk-1',
…
]我的池名解析只匹配仓库里那种排布,所以在 CI 上解析出零个池。两个用例因此挂掉: 上一次 run 为什么没抓到:
|
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
No blocking issues. LGTM! ✅
Deferred under the convergence posture (round 3, not a blocker) — recorded, not requested in this round:
scripts/tests/build-and-publish-image-workflow.test.js:360 — [review] stale test comment still describes the old 200-issue window
中文说明
无阻断问题。LGTM!✅
收敛姿态下延后(第 3 轮,非阻断)——已记录,本轮不要求修改:共 1 条(原文未翻译,列表见上方英文部分)。
— qwen3.8-max via Qwen Code /review (v0.22.3)
qqqys
left a comment
There was a problem hiding this comment.
APPROVE — 历史阻塞项全部在当前 head 核实修复,独立扫描无新 Critical
Independent re-review at head fb80b2704711a1f1475dad29f00be1ed9379d9d1。
历史阻塞项复核(在当前精确 head 上读码验证,非采信线程状态)
- R1-1(
report_failure权限块缺contents: 'read'→ 报告任务无法 checkout、先天死亡)— 已修复:workflow 的 job 级 permissions 现显式列出contents/actions/issues三键,并由update-ecs-runner-qwen-workflow.test.js(:170-175)按文本钉住。 - R1-2(4 个
runReport回放在test_windows车道必红)— 已修复:回放经replayable = platform !== 'win32' && jq门控(:236-238),且scripts/tests/vitest.config.ts的 win32 exclude 覆盖 bash 驱动的车道套件;YAML 断言在 Windows 照常运行。 - R1-6(legs 计数把 skipped 矩阵腿算进来 → resolve 失败时投出从未发生的池级断言;round-14 的
elif legs==0分支在真实 API 形状下不可达)— 已修复:gh api过滤器加select(.conclusion != "skipped")(ecs-fleet-update-failure-issue.sh :43),且 resolve-failure 夹具按真实 jobs API 形状生成五条全名 skipped 腿(test :91-96、:510),契约断言none was reached…与not.toContain('Verify version')钉住身体;在 027f5fc 上的正是本条的 CHANGES_REQUESTED 由此闭合。 - 顺带确认:R1-8(null body 触发 jq exit 5)虽为 Suggestion,也已在共享的
find-marked-issue.sh里以.body // ""+last(...)+ 全量 1000 列表收口。
当前 Critical-only 扫描
通读 123 行报告脚本、52 行去重 helper、workflow hunk 与 image-build 侧的共用改造:jobs-API 读取失败独立成形状(不猜测哪个门失败)、三形状 body 与 Target version: unresolved 互相自洽、去重失败降级不阻断投报(静默比重复更贵)、comment-not-rewrite 的重现语义、marker 客户端匹配的原因与 41-open-issue 窗口论证一致。测试实测:worktree 内 2 个套件 49/49 绿。未发现可证明的阻塞性正确性/安全/回归问题。
CI(非门禁,仅陈述):本 head 16 success / 27 skip,无失败无 pending。
✅ APPROVE — 三个历史 Critical 逐一在 head 代码与本地实测中确认闭合,当前扫描干净。
— 衍星 · read-only PR review (posted as qqqys)
|
Released in v0.23.0. |
Problem
A failed ECS fleet update produces no signal.
main-ci-failure-issue.ymlcannot cover it: that workflow watches test suites (E2E Tests,SDK Python,Qwen Code CI) onmain, whileUpdate ECS Runner Qwenruns offrepository_dispatch. The only trace is a red run in the Actions tab that nobody has a reason to open.The silence is what makes it expensive.
qwen-code-pr-review.ymlandqwen-triage.ymlinstall the CLI only when it is absent:On a self-hosted runner
/usr/bin/qwenalways exists, so a pool that misses an update keeps answering PRs on the old version indefinitely and is indistinguishable from a healthy one. On v0.22.3, 3 of the 4 pools failed and the split fleet ran for a full day before it was spotted by reading a version string in a review job's log.Change
A
report_failurejob that files — or comments on — a single issue naming the pools left behind. The step body is.github/scripts/ecs-fleet-update-failure-issue.sh, and the dedup lookup both failure reporters need is.github/scripts/find-marked-issue.sh.Decisions worth reviewing:
cancelledis not a failure. The per-pool concurrency group cancels an older dispatch's still-pending legs whenever a newer one arrives; that is routine and says nothing about the fleet. Onlyfailurereports.needs, and "some pool failed" is not actionable — "ecs-update-hk-1,ecs-update-hk-2are stale" is. The job-name prefix is a contract between the workflow'sname:template and the script's jq filter, so the suite derives both the fixture's pool names and the asserted prefix from the workflow itself: a rename on one side alone goes red instead of silently naming zero pools.resolvefailed and no pool was ever asked to install anything — the body no longer asserts a pool-level state that never happened, or points at aVerify versionstep that never ran. Reaching that shape means excludingskippedlegs from the count: a skipped matrix leg is still listed by the jobs API under its fully expanded name, so counting them would make the shape unreachable on exactly the runs it exists for. And "the job conclusions for this run could not be read", when the jobs API call itself fails — that branch names no shape at all, because which one failed is precisely what could not be read. The read captures its own exit status:set -eaborts on a failing command substitution feeding an assignment, so a transient 5xx would otherwise kill the reporter outright.scope/ci-cd. GitHub search tokenizes such a marker apart, so a search-based lookup never finds what the script files. Both labels are applied at creation, so the dedup key can never be half-written by a failed follow-upissue edit. The lookup is shared withimage-build-failure-issue.sh— same label space, same marker contract — so its two guards land once rather than twice: a.body // ""fallback (jq'scontains()exits 5 on thenullbody GitHub returns for a bodyless issue, which would abort the caller before it files anything), and a 1000-issue ceiling instead of a 200-issue newest-first window (this script's issue is opened once and only ever commented on, so it drifts toward the oldest slot and would eventually fall out and start filing duplicates; 41 openscope/ci-cdissues today). The match takes the oldest open marker-bearing issue, not the newest: the listing is newest-first and the marker is matched as a substring, so an issue that merely quotes it — a bug report about one of these reporters, say — would otherwise outrank the canonical one permanently. And a failing lookup degrades to "no known issue" rather than aborting: a rare duplicate costs less than the silence this job exists to break.main-ci-failure-issue.ymlalready documents forfile_issue.contents: 'read'is spelled out in the job'spermissions:block. A job-level block replaces the workflow-level one rather than merging with it, so without itactions/checkoutcannot fetch the ref and the job that exists to break the silence never runs.The step body lives in
.github/scripts/rather than inline, per the guidance in.github/workflows/.size-baseline. The workflow lands at 7187 bytes; its recorded baseline moves from 3303 — a number from before theresolvejob landed, which left 212 bytes of allowance — to the current size, in this PR, as that file asks.Testing
scripts/tests/update-ecs-runner-qwen-workflow.test.jsis 21 tests: 7 YAML-parse ones that run on every lane, and 14 bash replays behind the sameprocess.platform !== 'win32' && jqgatebuild-and-publish-image-workflow.test.jsuses. The replays need POSIX paths, a:-joined PATH and extensionless stubs, none of which the Windows lane can express; a whole-file exclude invitest.config.tswould have dropped the YAML-parse tests too.The
ghstub applies the script's real--jqfilter with realjq, so the pool-naming expression is exercised rather than mocked away, and it honours--limitonissue list, so the dedup window is a real fixture rather than a vacuous one.--label scope/ci-cdpresent on thecreatecall itselfscope/ci-cdlabelnull, both as the only issue and ahead of the marked oneresolvefailure without inventing a pool-level state, against a fixture carrying the skipped legs a real one producesname:template, and the shared lookup against both callerscontents: read+actions: read+issues: write, checkout before the script, and anifthat does not treatcancelledas a failureMutation matrix, 16 mutants, 16 killed (each reds exactly the test named for it). Round 1: drop
contents: 'read'; rename the matrix job'sname:template; put the lookup back on a 200-issue window; drop the.body // ""guard; swallow the jobs-API exit status; drop the${VERSION:-unresolved}default; fold the resolve-only body shape back into the pool one; point the sibling reporter back at an inline lookup; drop the no-conclusion fallback text. Round 2: countskippedlegs again; restore the pool-level headline on the unread branch; unguard either lookup; revertlast(tofirst((each suite separately); drop theif-block SC2016 directive (5 notes reappear under the repo's own shellcheck invocation).The replay gate's own mechanism, verified on a non-Windows host by flipping its predicate to
process.platform !== 'darwin':Tests 7 passed | 12 skipped (19)— every replay skipped, every YAML-parse test still running.actionlintpasses,shellcheck --check-sourced --enable=all --exclude=SC2002,SC2129,SC2310 --severity=stylereports nothing new on the three scripts, and the size ratchet passes.中文说明
问题
ECS fleet 更新失败时没有任何信号。
main-ci-failure-issue.yml覆盖不到它:那个 workflow 盯的是main上的测试套件(E2E Tests、SDK Python、Qwen Code CI),而Update ECS Runner Qwen是repository_dispatch触发的。唯一的痕迹就是 Actions 页面里一次红色的 run,而没人有理由去点开它。真正的代价来自这种「静默」。
qwen-code-pr-review.yml和qwen-triage.yml都是「不存在才安装」:self-hosted runner 上
/usr/bin/qwen一直都在,所以漏掉一次更新的池会无限期地用旧版本继续回答 PR,而且和健康的池看起来一模一样。v0.22.3 那次 4 个池挂了 3 个,割裂的 fleet 跑了整整一天,最后是靠在某个评审 job 的日志里读到版本号才发现。改动
新增
report_failurejob:开一个 issue(或在已有的那个上追加评论),点名哪些池被落下了。step body 在.github/scripts/ecs-fleet-update-failure-issue.sh,两个失败上报脚本共用的去重查找在.github/scripts/find-marked-issue.sh。几个值得 review 的决定:
cancelled不算失败。 每池的 concurrency group 会在新 dispatch 到来时取消旧 dispatch 中尚未开始的 leg,这是常规行为,与 fleet 状态无关。只有failure才上报。needs里拿不到;而且「有池失败了」不可操作,「ecs-update-hk-1、ecs-update-hk-2是旧版」才可操作。job 名前缀是 workflow 的name:模板与脚本 jq 过滤器之间的契约,所以测试里的 fixture 池名和被断言的前缀都从 workflow 本身推导:只改一边会直接变红,而不是在生产里静默地一个池都点不出来。resolve失败、矩阵根本没起来时说「run 在任何池被要求安装之前就失败了」——不再断言一个从未发生的池级状态,也不再指向一个从未跑过的Verify versionstep。要真正走到这一形态,必须把skipped的 leg 排除在计数之外:被跳过的矩阵 leg 仍会被 jobs API 以完全展开的名字列出,把它们算进去会让这个形态恰好在它为之存在的那些 run 上不可达。jobs API 本身调不通时说「本次 run 的 job conclusion 读不到」——这个分支不点名任何形态,因为「是哪一种失败了」正是读不到的东西。这次读取会显式接住自己的退出码:set -e确实会在「命令替换失败并喂给赋值」时中止,否则一次瞬时 5xx 会直接把上报器打死。scope/ci-cd收窄。GitHub 搜索会把这种 marker 切碎,基于搜索的查找永远找不到脚本自己开的 issue。两个 label 都在创建时打上,去重键不会因为后续issue edit失败而只写了一半。这个查找与image-build-failure-issue.sh共用——同一个 label 空间、同一套 marker 契约——所以它需要的两处加固只落一次:.body // ""兜底(GitHub 对没有正文的 issue 返回null,jq 的contains()在 null 上以 5 退出,会在任何东西被写出去之前打死调用方),以及把 200 条「最新优先」的窗口换成 1000 条的天花板(本脚本的 issue 只开一次、之后只追加评论,会一路漂到最旧的位置,迟早滑出窗口开始开重复 issue;今天有 41 个 open 的scope/ci-cdissue)。匹配取的是最旧的那条带 marker 的 open issue 而不是最新的:列表是最新优先、marker 是子串匹配,所以一个只是引用了 marker 的 issue(比如一份关于这两个上报器的 bug 报告)否则会永久压过 canonical issue。而查找本身失败时会降级为「没有已知 issue」而不是中止:一个罕见的重复 issue,代价远小于这个 job 存在就是为了打破的静默。main-ci-failure-issue.yml里file_issue已经写明的理由相同。contents: 'read'明确写出来。job 级的permissions:块是替换而不是合并 workflow 级的,缺了它actions/checkout拉不到 ref,这个「为打破静默而存在」的 job 自己就永远不会跑起来。step body 按
.github/workflows/.size-baseline的指引放进了.github/scripts/而非内联。workflow 最终 7187 字节;它记录的基线从 3303(resolvejob 落地之前的数字,只剩 212 字节余量)在本 PR 内更新为当前大小,正是那个文件要求的做法。测试
scripts/tests/update-ecs-runner-qwen-workflow.test.js共 21 个用例:7 个纯 YAML 解析用例在所有 lane 上都跑,14 个 bash 回放用例挂在与build-and-publish-image-workflow.test.js相同的process.platform !== 'win32' && jq门后面。回放需要 POSIX 路径、:拼接的 PATH 和无扩展名的桩,Windows lane 都表达不了;而在vitest.config.ts里整文件排除会把 YAML 解析用例一起丢掉。gh桩会用真的jq应用脚本里真实的--jq过滤器,所以点名池子的那段表达式是被真正执行的;桩还会遵守issue list上的--limit,所以去重窗口是个真 fixture 而不是空转。--label scope/ci-cd出现在create调用本身上scope/ci-cdlabel 的无关 issuenull的同 label issue:既作为唯一一条,也作为排在 marker 之前的一条resolve失败时不虚构池级状态,且 fixture 带着真实情况下会出现的 skipped legname:模板上,把共享查找钉在两个调用方上contents: read+actions: read+issues: write、脚本前先 checkout、以及if不把cancelled当失败变异矩阵 16 个变异体、16 个全杀(每个都只打红为它写的那个用例)。第 1 轮:删掉
contents: 'read';重命名矩阵 job 的name:模板;把查找退回 200 条窗口;删掉.body // ""守卫;吞掉 jobs API 的退出码;删掉${VERSION:-unresolved}默认值;把 resolve-only 的 body 形态折回池形态;把兄弟脚本指回内联查找;删掉「没有 conclusion」的兜底文案。第 2 轮:重新把skippedleg 计入;把池级首句放回「读不到」分支;分别去掉两个查找的守卫;把last(退回first((两个套件各一次);删掉if块上的 SC2016 指令(在仓库自己的 shellcheck 调用下重现 5 条 note)。回放门本身的机制,在非 Windows 宿主上把判据翻成
process.platform !== 'darwin'验证:Tests 7 passed | 12 skipped (19)——全部回放被跳过,全部 YAML 解析用例照跑。actionlint通过,shellcheck --check-sourced --enable=all --exclude=SC2002,SC2129,SC2310 --severity=style在三个脚本上没有新增报告,size ratchet 通过。