perf(ci): make the autofix fleet caps operator-tunable and raise them - #8731
Merged
Conversation
The three caps that bound the autofix review loop were compiled-in literals sized for a much smaller takeover pool, so growing the pool meant editing the workflow, opening a PR and waiting for review every time. The takeover pool is not static — it grew to 37 PRs by 2026-08-08, and it keeps growing. Back all three with repository variables, keeping the literals as fallbacks, so the loop is resized in Settings → Variables with no code change: QWEN_AUTOFIX_MAX_PARALLEL (fallback 20, was 5) QWEN_AUTOFIX_MAX_TARGETS_PER_SCAN (fallback 30, was 10) QWEN_AUTOFIX_MAX_CANDIDATE_INSPECTIONS (fallback 60, unchanged) Verified on a live runner that `max-parallel` accepts the expression and schedules by it — a 6-leg matrix resolving to 3 started exactly 3 legs and began the 4th only after a slot freed. Not assumed: an invalid expression here makes the whole file invalid, which this repository just paid 12.9 hours of dead review automation for. The raised fallbacks are sized against measurements, not guesses. At 5 slots the fleet served ~14% of the takeover pool at once, reproducing at a larger scale the 81-minute tail measured back at 3. The ecs-qwen fleet is 84 runners, so 20 concurrent legs take under a quarter of it, and the legs sampled that day finished in 3-28 minutes. Worst case rises to 100 runner-hours across the fleet (20 slots x the 300-minute job cap), and per-PR head-write concurrency groups are per-PR, so this adds no push contention. MAX_TARGETS_PER_SCAN has to stay above max-parallel or the scan cannot emit enough legs to fill the matrix. That relation is pinned for the fallbacks by an existing test and stated at both definitions for the variables, where it becomes an operator invariant. Mutation-tested, 4 of 4 caught: fallback equal to the budget, fallback above it, and dropping either variable back to a literal.
Collaborator
Contributor
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
yiliang114
approved these changes
Aug 8, 2026
yiliang114
left a comment
yiliang114
left a comment
Collaborator
There was a problem hiding this comment.
LGTM. The one thing I wanted to verify was the fallback path, and it checks out: the expression docs say fromJSON converts "any data type that can be represented in JSON or JavaScript", so the numeric || 20 default is valid when the variable is unset — which is the live case, none of the three variables are set in this repo yet. The targets-per-scan > max-parallel invariant stays pinned by tests for the fallback pair, and the sizing math (20 legs < a quarter of the 84-runner fleet, worst case 100 runner-hours under the 300-minute cap) is stated with its measurements.
One non-blocking note for operators: a non-numeric QWEN_AUTOFIX_MAX_PARALLEL will fail the review-address job at expansion — loudly, not silently, which is the right failure mode after #8720. Nothing blocks merge.
Collaborator
|
Thanks for the PR! Template looks good ✓ Problem: observed, not theoretical. The queueing tail the old caps produced is already documented in the workflow itself (the 81-minute slot wait measured back at Direction: aligned. This is the project's own autofix fleet — sizing the caps against measured fleet data (84 runners, 3–28 min legs) and making them tunable without a code change is the right follow-up to the #8720 outage, and the fallbacks keep an unconfigured repo correctly bounded. Size: not applicable — no core paths touched. 48 production lines ( Approach: scope feels right. All three cap definition sites are converted (I checked there are no others — the bash usage sites read the resolved env), the integer cap correctly uses Risk: no elevated risk signals — neither changed file matches the revert-correlated paths. Moving on to code review. 🔍 中文说明感谢贡献！ 模板完整 ✓ 问题：已观测到，不是理论问题。旧上限造成的排队尾部已经记录在 workflow 注释里（ 方向：对齐。这是项目自己的 autofix fleet——按实测数据（84 台 runner、单腿 3–28 分钟）确定上限，并使其无需改代码即可调整，是 #8720 故障之后正确的后续动作；兜底值也保证未配置的仓库仍在边界之内。 规模：不适用——未触及核心路径。生产代码 48 行（ 方案：范围合理。三个上限的定义点全部转换（确认没有其他定义点——bash 使用点读取的是解析后的 env）；整数上限正确地用了 风险：无升级风险信号——两个改动文件均未命中与 revert 相关的高风险路径。 进入代码审查 🔍 — Qwen Code · qwen3.8-max Reviewed at |
qqqys
approved these changes
Aug 8, 2026
Collaborator
|
Post-merge review note (no action needed on the diff itself — reviewed at Three things worth recording, one of which is a process gap rather than a code defect. 1. The test that guards this change never ran in CI. No I verified them out-of-band against the exact merged lines, and all six pass: The specific hazard I was checking for did not materialise: the two loosened whole-file regexes ( The gap is that a PR touching only 2. The 3. On the 5 → 20 raise itself I have no objection — the reasoning is data-backed (37 labelled PRs that day, sampled legs finishing in 3–28 minutes, under a quarter of the 84-runner fleet). The one thing I would keep an eye on is the interaction of 20 concurrent legs with the 300-minute job cap on a shared self-hosted fleet: the worst case reserves 100 runner-hours, and a backlog landing during a busy CI period is the scenario where regular PR legs would feel it. 中文说明合并后审查记录（diff 本身无需处理——审查提交 1. 守护本次改动的测试从未在 CI 中运行。 PR head 我在外部对照合并后的确切行做了验证，六项断言全部通过（parallel=20、targetBudget=30、不变式 20 < 30 成立）。 我重点检查的隐患没有发生：两条被放宽的全文件正则仍然匹配到 YAML 键而非新增散文注释——尽管本 PR 新增的注释里现在确实出现了 真正的缺口在于：只改动 2. 一旦运维设置了变量， 3. 至于 5 → 20 的提升本身我没有异议，其论据有数据支撑（当日 37 个带标签 PR、抽样任务 3–28 分钟完成、占 84 台 runner 不足四分之一）。唯一值得持续观察的是：20 条并行 leg 与 300 分钟 job 上限在共享自建 fleet 上的相互作用——最坏情况会占用 100 runner-小时，而积压恰好落在 CI 繁忙期时，常规 PR 任务会最先受影响。 |
Collaborator
|
Released in v0.21.8. |
qwen-code-dev-bot
added a commit
to qqqys/qwen-code
that referenced
this pull request
Aug 10, 2026
…M#8810) * perf(ci): make the triage budget operator-tunable and raise it The triage job's fixed 30-minute cap was killing every substantial PR's triage at exactly the budget — five observed (8723, 8726, 8731, 8764, 8801), each a full agent run discarded — and the successful-run distribution shows why: 22 jobs sampled ran median 5.8m / p90 22.3m / max 29.5m. A max just under the cap is a distribution being clipped, not one that fits. Back the budget with QWEN_TRIAGE_TIMEOUT_MINUTES (fallback 60), the same pattern as the autofix fleet caps in QwenLM#8731 and probe-verified there for expression-valued job fields. Triage is advisory: the worst case a longer budget risks is one held runner, while a killed run wastes the same runner and the work it was about to publish. Pinned by a test asserting the variable-backed form with its fallback; reverting to the fixed literal fails it. * Address review: sanitize the knob, pin the active line Both findings verified before fixing. The knob was unvalidated, and the failure mode is nasty precisely because the knob exists to be changed WITHOUT a PR: a malformed repository variable — '60 minutes', '1h', a 6O typo, quotes, a decimal — would make fromJSON throw at expression time, killing the triage job before any step runs: no always() finalize, no status comment, a silent stop whose run log blames fromJSON rather than the variable. And '0' parses fine into an instantly-cancelled job. timeout-minutes cannot be validated by the job that consumes it, so the authorize job (which triage already needs, and which can run bash) sanitizes it: bare-integer check, floor 10, ceiling 600, every rejection warning with the VARIABLE's name. The triage job consumes the sanitized output; || 60 covers the events where authorize is skipped. The pin was an unanchored substring over raw YAML, satisfied by a commented-out line while the job silently inherits GitHub's 360-minute default — the review probe-verified that mutant surviving. The pin now matches active lines only, and the sanitize step is replayed verbatim against the review's whole enumeration: unset, sane, zero, runaway, and five malformed shapes, each asserted to fall back and name the knob. Mutation-tested, 3 of 3 caught: the timeout line commented out, the floor clamp dropped, the integer check bypassed. * fix(ci): clamp over-long triage budgets before bash arithmetic (QwenLM#8810) A repository variable with more than 18 digits overflows bash's 64-bit arithmetic, and the wrapped residue can land inside [10,600], silently skipping both clamps with no warning — the one malformed shape where the sanitizer's diagnostics go dark. Check the digit count before the arithmetic and clamp such values to the ceiling with a warning. Also pin the seams the replay test cannot see: which repository variable feeds RAW, that the sanitize step is unconditional, the exact clamp boundaries, the 10# decimal guard, and the overflow clamp. * fix(ci): strip leading zeros from triage budget before the width guard (QwenLM#8810) * fix(ci): pin triage timeout seams on parsed YAML and runner shell flags (QwenLM#8810) --------- Co-authored-by: verify <verify@local> Co-authored-by: qwen-code-dev-bot <qwen-code-dev@service.alibaba.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Backs the three autofix fleet caps with repository variables and raises their defaults, so the loop is re-sized in Settings → Variables instead of by editing the workflow.
QWEN_AUTOFIX_MAX_PARALLELQWEN_AUTOFIX_MAX_TARGETS_PER_SCANQWEN_AUTOFIX_MAX_CANDIDATE_INSPECTIONSWhy it's needed
The takeover pool is not static — 37 PRs carried
autofix/takeoveron 2026-08-08 and the number keeps growing. Every previous resize meant editing the workflow, opening a PR, and waiting for review. Making the caps operator-tunable removes that loop entirely; the literals stay as fallbacks so an unconfigured repository is still correctly bounded.The defaults are raised because the old ones were sized for a much smaller pool:
max-parallel: 5served ~14% of the takeover pool at a time, reproducing at a larger scale the tail that was measured back when it was 3 (a scan selecting 7 PRs left the 7th waiting 81 minutes for a slot).MAX_TARGETS_PER_SCAN: 10emitted at most 27% of the eligible set per tick, deferring the rest by a full scan every time. It also has to stay abovemax-parallel, or the scan cannot emit enough legs to fill the matrix.Sized against measurements rather than a guess: the
ecs-qwenfleet is 84 runners, so 20 concurrent legs occupy under a quarter of it, and the executed legs sampled that day finished in 3–28 minutes. Worst case rises to 100 runner-hours across the fleet (20 slots × the 300-minute job cap). Per-PR head-write concurrency groups are per-PR, so raising this adds no push contention.max-parallelaccepting an expression was verified, not assumedAn invalid expression here makes the entire workflow file invalid — the failure mode this repository just paid 12.9 hours of dead review automation for (#8720). So it was measured on a live runner before being written: a 6-leg matrix whose
max-parallelresolved throughfromJSON(vars.X || 3)That also confirms the
|| fallbackpath works when the variable is unset, which is how this ships.Reviewer Test Plan
How to verify
Expected: 122/122. Full scripts suite: 50 files, 1053 passed.
.github/scripts/qwen-triage-workflow.test.mjs: 56/56.yamllintclean.To exercise a knob after merge, set
QWEN_AUTOFIX_MAX_PARALLELin Settings → Variables and watch the next scan's matrix; no deploy is involved.Evidence (Before & After)
N/A for UI. Before/after is the cap table above plus the probe scheduling trace.
Mutation-tested — 4 of 4 caught:
max-parallelvariable back to a literalMAX_TARGETS_PER_SCANvariable back to a literalThe existing invariant test — bounds fleet-wide simultaneity below the per-scan target budget — was kept meaningful by pointing it at the fallback pair, so an unconfigured repo is still provably bounded. For configured repos the relation becomes an operator invariant, stated at both definitions.
Tested on
Risk & Scope
QWEN_AUTOFIX_MAX_PARALLELback down without a code change.MAX_CANDIDATE_INSPECTIONSis variable-backed but its value is unchanged: the same-repo candidate pool was 51, still under 60.Linked Issues
Follow-up to the #8648 / #8720 review outage: draining that backlog is what made the fleet caps the next bottleneck.
中文说明
What this PR does
把 autofix 的三个 fleet 上限改为由仓库变量提供，并提高其默认值，这样以后在 Settings → Variables 里调整即可，不必再改 workflow。
QWEN_AUTOFIX_MAX_PARALLELQWEN_AUTOFIX_MAX_TARGETS_PER_SCANQWEN_AUTOFIX_MAX_CANDIDATE_INSPECTIONSWhy it's needed
takeover 池并不是静态的——2026-08-08 有 37 个 PR 带着
autofix/takeover标签，而且还在增长。此前每一次调整都意味着改 workflow、开 PR、等评审。把这些上限交给运维变量之后，这个循环就彻底消失了；字面量保留为兜底值，因此未做任何配置的仓库依然处在正确的边界之内。默认值之所以要提高，是因为原值是按远小于当前规模的池子设定的：
max-parallel: 5同一时刻只服务 takeover 池的约 14%，把当年在 3 时测到的尾部效应在更大规模上重演了一遍（一次选中 7 个 PR 的 scan，第 7 个等了 81 分钟 才拿到槽位）。MAX_TARGETS_PER_SCAN: 10每个 tick 最多只能 emit 合格集合的 27%，其余每次都被顺延一整个 scan。它还必须高于max-parallel，否则 scan 根本 emit 不出足够的腿来填满矩阵。数值是按实测而非猜测确定的：
ecs-qwen池共 84 台 runner，因此 20 条并发腿占用不到四分之一；当天采样到的实际执行腿耗时在 3–28 分钟。最坏情况上升到整个 fleet 100 runner-小时（20 槽 × 300 分钟的 job 上限）。per-PR 的 head-write 并发组是按 PR 划分的，所以提高该值不会增加 push 争用。max-parallel能否接受表达式是实测的，不是假设这里写错一个表达式会让整个 workflow 文件失效——本仓库刚刚为这种故障付出了 12.9 小时评审自动化全停的代价（#8720）。所以在落笔之前先在真实 runner 上量过：一个 6 条腿的矩阵，其
max-parallel经由fromJSON(vars.X || 3)求值这同时验证了变量未设置时
|| 兜底值这条路径是有效的，而本 PR 正是以这种形态发布的。Reviewer Test Plan
How to verify
预期 122/122。scripts 全量套件：50 个文件、1053 passed。
.github/scripts/qwen-triage-workflow.test.mjs：56/56。yamllint干净。合入后想验证某个旋钮，在 Settings → Variables 里设置
QWEN_AUTOFIX_MAX_PARALLEL，然后观察下一次 scan 的矩阵即可，全程不涉及任何部署。Evidence (Before & After)
界面部分 N/A。Before/After 即上面的上限对照表，加上探针的调度轨迹。
变异测试 —— 4 个全部被捕获：
max-parallel变量退回字面量MAX_TARGETS_PER_SCAN变量退回字面量既有的不变量测试——bounds fleet-wide simultaneity below the per-scan target budget——通过指向兜底值这一对而继续有效，因此未做配置的仓库依然可证明是有界的。对已做配置的仓库，该关系转为运维不变量，并在两处定义点均已写明。
Tested on
Risk & Scope
QWEN_AUTOFIX_MAX_PARALLEL即可，无需改代码。MAX_CANDIDATE_INSPECTIONS虽已变量化但取值未变：同仓候选池为 51，仍在 60 以内。Linked Issues
#8648 / #8720 评审停摆的后续：正是在消化那批积压时，fleet 上限成为了下一个瓶颈。