ci(autofix): recover from generated-artifact CI gates and stop silent stalls - #6998
Merged
Conversation
… stop silent stalls The autofix bot stalled on PRs that edit settingsSchema.ts without regenerating settings.schema.json (e.g. #6984): a CI-only freshness gate it could neither run, see, nor recover from. - Give the agent the tool and the instruction to regenerate: add `npm run generate:settings-schema` to the develop-issue and address-review coreTools allowlists, and a SKILL rule to regenerate + commit a source's generated artifact. - Mirror CI's "Check settings schema is up-to-date" step in both verify gates so a stale artifact fails locally instead of red-on-CI after push. - Inject the actual failing STEP name + a log excerpt into feedback.md so the agent diagnoses from the real failure instead of guessing from local test runs. SKILL now forbids "pre-existing"/environment excuses without evidence. - Decouple the feedback watermark from base-sync pushes: use the last eval marker (what the agent evaluated), not the head commit date, so an "Update branch" merge can no longer bury unaddressed maintainer feedback. Use PR createdAt as the pre-first-eval floor. - Bound the pending-check skip so a check wedged pending can't strand a PR forever; always post a handoff comment + eval marker on failure so the loop never goes silent; add an issue_comment trigger so an @-mention from a trusted maintainer re-triggers the review pass promptly.
Collaborator
|
Thanks for the PR, @wenshao! Template looks good ✓ — all required sections present with bilingual content. Problem: Observed bug with concrete evidence. The autofix bot stalled on #6984 when a settings schema source edit left Direction: Aligned. This is CI infrastructure hardening — closing gaps in the autonomous loop that currently require human intervention when they shouldn't. No product-direction concerns. Size: Not applicable — no core modules touched. Changes are confined to Approach: The five coordinated changes feel right-sized for the problem. Each addresses a specific failure mode identified in the motivating incident, and the scope note explicitly respects existing design decisions (no comment-trigger path, no raw-log injection). No scope creep. The verify-gate schema check adds one Moving on to code review. 🔍 中文说明感谢贡献，@wenshao！ 模板完整 ✓ — 所有必要章节齐全，含双语内容。 问题： 已观测到的 bug，有具体证据。autofix bot 在 #6984 上卡死：settings schema 源码改动导致 方向： 对齐。这是 CI 基础设施加固——闭合自主循环中本不该需要人工介入的缺口。无产品方向顾虑。 规模： 不适用——未触及核心模块。改动限于 方案： 五处协同改动与问题匹配。每处针对触发事件中的具体失败模式，范围说明明确遵守既有设计决定（不新增评论触发路径、不注入原始日志）。无范围蔓延。verify gate 的 schema 检查每次运行增加一次 进入代码审查 🔍 — Qwen Code · qwen3.7-max Reviewed at |
Collaborator
Code ReviewIndependent proposal: before reading the diff, my approach would have been:
The PR's solution matches all four points and goes further with the Correctness: No issues found. The key changes are well-reasoned:
Reuse: No new utilities or abstractions. The schema check reuses the existing Style: Follows the workflow's existing patterns — consistent comment prefixes, VerificationReal-scenario testing: This is a non-UI CI workflow change — tmux testing is not applicable. The contract test suite (48 assertions) is the appropriate verification mechanism: it validates the workflow YAML structure, watermark logic, handoff behavior, and all the behavioral contracts this PR changes. All pass. No Critical or blocking findings. The implementation is clean, well-tested, and addresses a real gap. 中文说明代码审查独立方案： 在读 diff 之前，我的方案是：
PR 的方案覆盖了以上四点，还额外做了 正确性： 未发现问题。关键改动均有理有据：
复用： 无新工具或抽象。schema 检查复用了现有的 风格： 遵循工作流既有模式——一致的评论前缀、 验证真实场景测试： 这是非 UI 的 CI 工作流改动——tmux 测试不适用。契约测试套件（48 条断言）是合适的验证方式：验证了工作流 YAML 结构、水位线逻辑、交接行为以及本 PR 改动的所有行为契约。全部通过。 无 Critical 或阻塞性问题。实现干净、测试充分、解决了真实缺口。 — Qwen Code · qwen3.7-max Reviewed at |
Collaborator
|
Confidence: 5/5 Self-audit: re-read the full diff twice. The watermark change is the linchpin — it correctly decouples "what the agent has seen" from "when the branch was last touched," and the motivating replay data proves the fix. The four other changes are each independently sound and mutually reinforcing. Tests cover every behavioral change. No gaps found. Approving. ✅ 中文说明置信度：5/5 自审：完整 diff 通读两遍。水位线改动是关键——正确地将"agent 已看到的内容"与"分支最后被触碰的时间"解耦，动机中的回放数据证明了修复效果。其余四处改动各自独立可靠且相互增强。测试覆盖了所有行为变更。未发现缺口。 通过 ✅ — Qwen Code · qwen3.7-max Reviewed at |
qwen-code-ci-bot
previously approved these changes
Jul 16, 2026
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
LGTM, looks ready to ship. ✅
Contributor
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
…thin existing safety guards Respects two deliberate, tested design decisions that the first cut collided with (caught by scripts/tests/qwen-autofix-workflow.test.js): - Drop the issue_comment @-mention trigger and its route branch: the workflow intentionally does not expose comment-triggered autofix (only pull_request_review:submitted) to avoid redundant runs and comment-command surface. The scheduled scan plus the watermark fix already re-target a PR after maintainer feedback. - Drop the raw CI-log injection into feedback.md: feedback fed to the model is deliberately sanitized and must not pull in URLs / raw context (a prompt-injection surface). Keep only the sanitized check-name rendering (.name // .workflowName, still gsub+truncated). Update the assertions that the retained improvements (watermark decoupling, pending-check staleness bound, always-post-handoff-on-failure) legitimately changed. Workflow test: 48/48 green.
Collaborator
Author
|
@qwen-code /triage |
qwen-code-ci-bot
previously approved these changes
Jul 16, 2026
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
LGTM, looks ready to ship. ✅
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed. Suggestions are inline.
— qwen3.7-max via Qwen Code /review
…over schema commands Addresses the three inline /review suggestions on the PR: - Handoff on any non-success end, not just "failure". A 120-minute job-timeout cancellation sets job.status = "cancelled", which the "== failure" check missed — leaving no marker and no comment, so the next scan re-targeted the same feedback with the same round (an invisible loop). Use "!= success"; the step only runs on failure()/cancelled()/dry-run and dry-run is excluded. - Fold the createdAt fetch into the existing statusCheckRollup gh pr view call (statusCheckRollup,createdAt), removing one GitHub API round-trip per PR scanned. - Add test coverage the earlier diff lacked: assert generate:settings-schema is in both agent allowlists and that both verify gates run the schema-freshness check, so a future edit can't silently drop the guard this PR adds.
Collaborator
Author
|
Thanks — all three suggestions addressed in
Workflow contract test stays green (48/48). |
qqqys
previously requested changes
Jul 16, 2026
qqqys
left a comment
qqqys
left a comment
Collaborator
There was a problem hiding this comment.
Not reviewed: coverage — could not read the agents' transcripts (the CLI did not export QWEN_CODE_PROJECT_DIR / QWEN_CODE_SESSION_ID, so this run cannot find the harness's record of what its agents did), so this run cannot show that any of the diff was read.
— GPT-5 via Qwen Code /review
…al + 2 suggestions) Addresses the CHANGES_REQUESTED review of the handoff (E-4) and pending-check staleness (E-3) logic: - Suppress the handoff once a run published a result (OUTCOME fixed/noop), so a later always() step failing the job (e.g. artifact upload) can no longer post a contradictory acted=false handoff over a reported success. - Bound the agent step at 80m, well under the 120m job timeout, so a runaway agent fails the STEP (not the job) and the always() report step still runs and hands off — a job-level timeout would cancel that step too and go silent. - On a pre-prepare crash (empty NEWEST) the watermark can't advance, so write a terminal marker (round = MAX_ROUNDS) and skip on the highest marker round (not last-by-ts), so the scan stops re-handing-off instead of repeating until MAX_ROUNDS. - Raise the pending-staleness bound from 30m to 240m so an active check (review-pr ~50m, review-address up to 120m) is never aged out mid-flight and the same feedback double-processed; only truly-dead checks are ignored. - Prefer the agent's detailed failure.md over the generic handoff.md wrapper. Adds a bash-replay test that extracts the actual POST_HANDOFF decision and MARK_ROUND logic from the workflow and exercises the state transitions (published+late-failure, dry-run, verify failure, pre-verify crash, cancellation; terminal vs incremental round). Workflow test: 49/49.
Collaborator
Author
|
Thanks @qqqys — all four Critical findings and both suggestions addressed in Critical
Suggestions
Workflow contract test now 49/49. |
Collaborator
Author
Review — verified against the PR-head contract test and live GitHub dataOverview. Four coordinated fixes to the autofix loop, and the PR body describes them accurately: (A) a deterministic settings-schema freshness gate in both verify steps, (C) the generator added to both agent allowlists plus SKILL.md rules, (D) eval-marker-only watermark with a PR-creation floor, (E) always-visible failure handoff with terminal-round parking — plus stale-pending-check aging and a step-level 80-minute timeout that keeps the How I verified
Findings, ranked
Verdict: the four fixes are real, mutually consistent, and each is backed by an executed test or a live-data replay; every analyzed scenario strictly improves on main except the transient-failure case in finding 1, which trades main's silent-but-self-healing retry for a visible-but-permanent stop. I'd decide finding 1 before merge (a one-line change if accepted); 2, 3, and 5 are cheap hardening; nothing else blocking. 中文版评审 — 已对照 PR 分支契约测试与 GitHub 线上数据验证概述。 对 autofix 回路的四处协同修复,PR 描述准确:(A)两个 verify 步骤中的确定性 schema 新鲜度门禁;(C)两处 agent 白名单加入生成命令 + SKILL.md 规则;(D)水位线改为仅用 eval 标记、以 PR 创建时间为地板;(E)失败必留可见交接 + 终局轮次停靠;另有 pending 检查的过期老化和 80 分钟步级超时(保证 验证方式
发现(按严重度)
结论: 四处修复真实、相互自洽,均有已执行的测试或线上数据回放背书;所有分析场景相对 main 严格更优,唯一例外是发现 1 的瞬态失败情形——用"可见但永久的停止"换掉了 main"静默但可自愈的重试"。建议合并前对发现 1 做个决定(若采纳是一行改动);2、3、5 是廉价加固;其余不阻塞。 |
yiliang114
reviewed
Jul 16, 2026
yiliang114
left a comment
yiliang114
left a comment
Collaborator
There was a problem hiding this comment.
The follow-up commits look solid — PENDING_STALE_MIN=240, the max(.round) calculation, the handoff guard for fixed/noop outcomes, and the step-level timeout-minutes: 80 all close real gaps. Two non-blocking suggestions inline.
…ter SKILL wording Two non-blocking review suggestions: - The issue-phase verify gate's schema-freshness check now writes outcome=failed before exit 1, matching the address-review gate, so the issue-phase step summary shows outcome=failed instead of outcome=unknown. - Reword the SKILL rule from "do not invent environment excuses" to "do not skip a failing check by attributing it to the environment without evidence," which keeps the intent (no hand-waving a real failure as an env issue) without discouraging the agent from reporting a genuine infra failure.
Collaborator
Author
|
Thanks @yiliang114 — both applied in
|
yiliang114
requested changes
Jul 16, 2026
yiliang114
left a comment
yiliang114
left a comment
Collaborator
There was a problem hiding this comment.
Not reviewed: coverage — could not read the agents' transcripts (the CLI did not export QWEN_CODE_PROJECT_DIR / QWEN_CODE_SESSION_ID, so this run cannot find the harness's record of what its agents did), so this run cannot show that any of the diff was read.
— GPT-5 via Qwen Code /review
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed. Suggestions are inline. Unresolved, please confirm: [Critical] .github/workflows/qwen-autofix.yml:1497 — whether the schema freshness check should run before the no-op early return (design tradeoff: agent judgment vs. structural guard) [Critical] .github/workflows/qwen-autofix.yml:1039 — whether the head-commit-date fallback when createdAt is unavailable recreates the watermark-burial bug in a narrow API failure scenario
— qwen3.7-max via Qwen Code /review
…ble floor, robust handoff
Review round 3 (2 critical + 3 suggestions):
- Run the review gate's settings-schema freshness check BEFORE the no-op/
unchanged return, so a stale-schema PR the agent wrongly no-ops fails
(outcome=failed) instead of being reported as evaluated while CI stays red —
the exact motivating bug. Single check now covers every path; ordering is
asserted in the test.
- Never fall back to the mutable head commit date for the pre-first-eval
watermark floor: if the PR metadata query fails, use an empty (over-inclusive,
never-buries) floor. A base-sync HEAD as the floor would recreate the burial
bug. Removes the now-unused HEAD_SHA lookup.
- Guard the terminal handoff marker's timestamp (MARK_TS=${NEWEST:-${WATERMARK:-unknown}})
so a cascading API failure that blanks WATERMARK can't emit an unparseable
`ts=` that the scan regex skips, defeating the terminal-round guard.
- Truncate failure.md through `iconv -f utf-8 -t utf-8 -c` so a byte-level
head -c can't split a multi-byte sequence and corrupt the comment body.
- A pre-prepare crash (empty NEWEST) now says "could not start evaluation"
instead of "round 5/5", which would imply MAX_ROUNDS attempts were made.
Workflow test: 49/49.
Collaborator
Author
|
Thanks — all addressed in Critical
Suggestions
Workflow test now 49/49 with the added ordering + invariant assertions. |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed. Suggestions are inline.
— qwen3.7-max via Qwen Code /review
Four review suggestions, test-only: assert the else-branch floor
(EFF_WM=${CREATED_WM}, not the old PUSH_WM), the staleness jq filter
(.startedAt // ... // $cut), the JOB_STATUS env declaration (else it is always
empty → over-eager handoffs), and the .name // .workflowName feedback format —
so a regression on any of these is caught rather than passing silently.
Collaborator
Author
|
Thanks — all four test-coverage gaps closed in
Test now 49/49 with these invariants pinned. |
Collaborator
Author
ReviewOverviewFour coordinated changes to the autofix loop, all correctly aimed at the #6984 stall class: (A) the deterministic verify gates now mirror CI's settings-schema freshness check, (C) the agent gets the regeneration command in its allowlist plus explicit prompt guidance, (D) the scan watermark becomes eval-marker-based with the immutable PR Verification performed
Findings (none blocking)1. 2. The pending-check 3. Review-gate schema regen can false-fail against a stale core dist. 4. The timeout arithmetic is plausible but unproven. 5. Pre-existing, follow-up material: the marker-spoof strip is bypassable. Quality notes
Verdict: solid, carefully reasoned fix for a real operational failure mode; the four changes compose coherently and the test coverage is unusually strong for CI shell logic. Findings above are suggestions and hardening follow-ups, not blockers. |
qwen-code-ci-bot
requested changes
Jul 16, 2026
Collaborator
Author
Re-review at
|
| # | Finding | Status |
|---|---|---|
| Blocker | --check reverted from main → gates fail-open after merge |
✅ Fixed — version-agnostic gate + in-file rationale + negative test assertion. |
| 2 | No-startedAt pending checks never block |
✅ Pinned in tests — the new jq behavioral replay executes the actual extracted filter with three labeled cases (recent start → blocks; stale start → aged out; queued/no startedAt → does not block). The trade-off is now executable documentation, which is better than the comment I asked for. |
| 3 | Terminal-marker recovery undocumented | ✅ Fixed in 1acbd08, unchanged here. |
| 4 | Misleading "crashed or timed out" fallback | ✅ Fixed in 1acbd08, unchanged here. |
| 1 | Review-address gate regenerates against the base-built core dist | Accepted residual — still not named in the gate comment, but it's a narrow edge (only bites when a bot PR changes a core export that feeds the schema and the agent never ran npm run build), and it fails toward a handoff, not a silent pass. Fine to ship as-is; no further action requested. |
New-code check (this delta)
I looked for fresh issues in the round-3 changes and found none worth raising: the jq-replay extraction regex anchors on the only --arg cut call site (and a wrong extraction would fail the toBe assertions rather than pass vacuously); execFileSync('jq', …) is safe on every environment that already runs this file's bash replays (GitHub-hosted runners ship jq); the restore-on-failure git checkout -- is correctly scoped to the tracked schema file.
Verification performed
- Contract test at
6e39e43in a clean detached worktree: 49/49 pass. - Grepped the head workflow for
--check/generate:settings-schema: all invocations plain, references to--checkare comment-only. - Confirmed the delta touches exactly
.github/workflows/qwen-autofix.ymlandscripts/tests/qwen-autofix-workflow.test.js; gates match CI's ci.yml step semantics plus the restore.
Verdict
Approve. The loop-hardening design from round 1 stands, the round-2 regression is cleanly reverted with a guard against recurrence, and the pending-staleness semantics are now behaviorally tested. One housekeeping note: the PR body still says the contract test passes 48/48 — it's 49/49 at head.
中文版(Chinese translation)
在 6e39e43 上的重审(接续第 1、2 轮)
阻塞项已解决,所有遗留发现均已修复或以测试形式钉住。LGTM。
阻塞项解决 — 已验证
schema gate 回到了与版本无关的 regenerate + git status --porcelain 对比(选项 A)。在 6e39e43 上验证:
- 两处 gate 调用均为普通
npm run generate:settings-schema;--check只出现在注释中——作为被记录的不可使用项,并写明 revert: remove local PR verification gate #7031 与 fail-open 失效模式。合并后的行为不再依赖 main 上 generator 的版本。 - 契约测试新增负向断言
not.toContain('generate:settings-schema -- --check'),被 revert 的参数依赖若再被引入会直接红测——正是第 2 轮要求的防漂移守卫。 - 比第 1 轮原始 gate 更进一步:失败路径上
git checkout -- "${SCHEMA_FILE}" || true在exit 1前恢复工作树,gate 失败不再留下脏的再生成产物。
此前发现 — 最终状态
| # | 发现 | 状态 |
|---|---|---|
| 阻塞 | --check 已从 main revert → 合并后 gate fail-open |
✅ 已修复——版本无关的 gate + 文件内说明 + 负向测试断言。 |
| 2 | 无 startedAt 的 pending 检查不再阻塞 |
✅ 已在测试中钉住——新的 jq 行为回放执行真实提取的过滤器,覆盖三种情形(新近启动→阻塞;超期→老化剔除;排队未启动→不阻塞)。该权衡现在是可执行的文档,比我要求的注释更好。 |
| 3 | 终局标记恢复方式未写明 | ✅ 1acbd08 已修复,本轮无变化。 |
| 4 | 误报"崩溃或超时"的兜底文案 | ✅ 1acbd08 已修复,本轮无变化。 |
| 1 | review-address gate 用 base 构建的 core dist 再生成 | 接受为残留——gate 注释仍未点名,但这是窄边缘(仅当 bot PR 改动影响 schema 的 core 导出且 agent 从未跑过 npm run build 时才触发),且失败方向是交接而非静默放行。可按现状合并,不再要求处理。 |
本轮新代码检查
对第 3 轮改动查找新问题,无值得提出的:jq 回放的提取正则锚定唯一的 --arg cut 调用点(提取错误会使 toBe 断言失败而非空过);execFileSync('jq', …) 在所有已运行本文件 bash 回放的环境上都可用(GitHub 托管 runner 自带 jq);失败恢复的 git checkout -- 正确限定在被跟踪的 schema 文件上。
已执行的验证
6e39e43干净 detached worktree 中契约测试 49/49 通过。- grep head 工作流中的
--check/generate:settings-schema:所有调用均为普通形式,--check仅存在于注释。 - 确认 delta 仅涉及
.github/workflows/qwen-autofix.yml与scripts/tests/qwen-autofix-workflow.test.js;gate 语义与 ci.yml 步骤一致并附恢复逻辑。
结论
批准。第 1 轮的回路加固设计成立,第 2 轮的回归被干净回退并加了防复发守卫,pending-staleness 语义现有行为级测试。一条整理性备注:PR 描述仍写契约测试 48/48,head 实际为 49/49。
qwen-code-ci-bot
requested changes
Jul 16, 2026
…ashes Both verify gates ran the generator unguarded: if it crashes (e.g. a type error the agent introduced in the schema source), set -eo pipefail aborts the step before outcome=failed is written, leaving OUTCOME unset (the handoff still fires via job.status, but the outcome is inferred rather than explicit). Wrap the generator in `if ! ...; then outcome=failed; exit 1; fi` so the failure is explicit and does not depend on the job.status fallback. Test asserts the guard.
Collaborator
Author
|
Thanks — both gates hardened in the latest commit: Wrapped the generator in 49/49. |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed — no blockers. Suggestions are inline.
— qwen3.7-max via Qwen Code /review
Collaborator
Author
|
@qwen-code /triage |
Address review: the 16-line schema-freshness gate (generator crash guard, porcelain check, restore, outcome=failed) was duplicated verbatim between the issue-fix verify step and the triage-and-address verify step, so an edit to either copy could silently diverge from the other. Move it to .github/scripts/check-settings-schema.sh and call it from both sites; the site-specific rationale comments stay at the call sites, the shared mechanics and the crash-guard rationale live in the script. The script preserves the exact step contract, verified with a PATH-shim harness over a temp git repo: generator ok + fresh schema exits 0 with no output written; a generator crash writes outcome=failed to GITHUB_OUTPUT and exits 1; a stale schema prints the diff, restores the file, writes outcome=failed, and exits 1.
…-check script The main-merge extracted the inline schema-freshness block into the shared .github/scripts/check-settings-schema.sh (both verify gates now invoke it), which broke the test that asserted the inline generate/porcelain strings in the step. Assert the step invokes the script and that the SCRIPT holds the logic (regenerate, crash guard, git status --porcelain, no --check, outcome=failed), and that the review gate's script call precedes the no-op return.
Collaborator
Author
|
@qwen-code /triage |
qwen-code-ci-bot
approved these changes
Jul 17, 2026
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
No issues found. LGTM! ✅
— qwen3.7-max via Qwen Code /review
Collaborator
|
Thanks for the PR! Template looks good ✓ Problem: observed bug — the autofix bot stalled on #6984 with clear evidence: settings schema source was edited without regenerating the committed artifact, CI's freshness gate went red, and the bot could neither fix it (command not in allowlist), detect it (invisible to build/typecheck/lint/vitest), nor report it (went silent). A maintainer's follow-up comments were also buried by a base-sync merge advancing the watermark past them. Direction: aligned — autofix loop reliability is core to the bot's mission. Closing these silent-stall failure modes directly improves the autonomous repair cycle that maintainers depend on. Size: ~267 production logic lines (additions + deletions across Approach: the four coordinated changes each close a specific, named gap in the failure loop — agent tooling (C), deterministic verify gate (A), watermark redesign (D), and handoff guarantee (E). The extracted Moving on to code review. 🔍 中文说明感谢贡献！ 模板完整 ✓ 问题：已观测到的 bug——autofix bot 在 #6984 上卡死，证据清晰：修改了 settings schema 源码却没有重新生成已提交产物，CI 新鲜度 gate 变红，而 bot 既修不了（命令不在白名单）、也检测不到（对 build/typecheck/lint/vitest 隐形）、也无法报告（静默失联）。维护者的后续评论还被基线同步合并推进水位线而埋掉。 方向：对齐——autofix 回路可靠性是 bot 使命的核心。闭合这些静默卡死的失败模式直接改善了维护者依赖的自主修复循环。 规模：约 267 行生产逻辑代码（ 方案：四处协同改动各自闭合失败回路中一个具体的、已命名的缺口——agent 工具（C）、确定性 verify gate（A）、水位线重设计（D）、交接保证（E）。提取的 进入代码审查 🔍 — Qwen Code · qwen3.7-max Reviewed at |
Collaborator
Code ReviewIndependent proposal before reading the diff: I'd have added The PR matches and exceeds that proposal. The extracted The watermark redesign is clean: The handoff decision matrix ( The SKILL.md additions are well-scoped: generated-artifact rule, CI failure diagnosis from evidence, and no environment-attribution-without-evidence rule. Each directly addresses the failure mode from #6984. No critical findings. No AGENTS.md violations. The code is straightforward, well-commented where the why is non-obvious (the VerificationContract test at One housekeeping nit: the PR body still says the contract test passes 48/48 — it's 49/49 at head. 中文说明代码审查读 diff 前的独立方案：把 PR 匹配并超越了该方案。提取的 水位线重设计干净： 交接决策矩阵在 SKILL.md 新增内容范围合理，每条都直接对应 #6984 的失败模式。 无阻塞性发现。无 AGENTS.md 违规。代码简洁，在 why 非显而易见的地方注释充分（ 验证
一条整理性备注：PR 描述仍写契约测试 48/48，head 实际为 49/49。 — Qwen Code · qwen3.7-max Reviewed at |
Collaborator
|
Confidence: 5/5 — clean across every stage, would merge without hesitation. This is a well-reasoned hardening of the autofix loop, motivated by a real stall incident (#6984) and backed by unusually strong test coverage for a workflow file. The four changes form a coherent fix: each closes a specific gap in the failure loop, and together they make the bot resilient to an entire class of silent-stall failures (stale generated artifacts) rather than just the one instance that triggered it. The PR went through three rounds of iterative review before this triage run, and every finding — including a blocking The watermark redesign is the standout change — switching from mutable push dates to the bot's own eval markers (with immutable Approval guardrail check: not a fork PR, not a refactor — no block. Approving. 中文说明Confidence: 5/5 — 各阶段均通过，毫不犹豫批准。 这是一次论证充分的 autofix 回路加固，由真实的卡死事件（#6984）驱动，对一个 workflow 文件而言测试覆盖异常强。四处改动构成一个连贯的修复：各自闭合失败回路中的一个具体缺口，合在一起让 bot 对一整类静默卡死失败（过期的生成产物）具备韧性，而不仅仅是触发它的那一个实例。 PR 在本次 triage 之前经历了三轮迭代审查，每一个发现——包括第 2 轮捕获的阻塞性 水位线重设计是亮点——从可变的 push 日期切换到 bot 自己的 eval 标记（以不可变的 批准守卫检查：非 fork PR，非 refactor——无阻塞。 批准。 — Qwen Code · qwen3.7-max Reviewed at |
qwen-code-ci-bot
approved these changes
Jul 17, 2026
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
LGTM, looks ready to ship. ✅
This was referenced Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Hardens the autonomous autofix loop (
qwen-code-dev-bot) so it can recover from a class of CI failures it currently gets permanently stuck on: a source edit whose committed generated artifact was not regenerated. The concrete trigger is a change to the settings schema source that leavessettings.schema.jsonstale — CI's "Check settings schema is up-to-date" step fails, but that failure is invisible to build/typecheck/lint/Vitest, so the bot could neither run the fix, catch it, nor avoid going silent. Four coordinated changes close that loop: the agent is now given the tool and the instruction to regenerate committed artifacts; the deterministic verify gate mirrors CI's freshness check so a stale artifact fails locally before push; the "have I already considered this feedback?" watermark no longer advances on unrelated base-sync merges (so a maintainer comment can't get buried); and a failed run always leaves a visible handoff + marker instead of stalling silently. A stuck pending check can also no longer strand a PR forever.Why it's needed
The autofix bot stalled on a real PR: it edited the settings schema source without regenerating
settings.schema.json, CI's freshness guard went red, and the bot (1) could not have fixed it even given the exact instructions, becausenpm run generate:settings-schemawas in neither the agent tool allowlist nor the verify gate; (2) after a maintainer commented the fix, a manual "Update branch" merge bumped the push-watermark past that comment, burying it so every subsequent scan reported "nothing new"; and (3) any run that failed before writing a handoff file left no comment and no eval marker at all, so the loop went silent and a human had to take over by hand. This is a recurring failure mode for any source with a committed generated artifact, so it is worth fixing structurally rather than per-PR.Scope note: this PR deliberately stays within two existing, tested design decisions — it does NOT add a comment-triggered run path (the workflow intentionally triggers only on
pull_request_review:submitted), and it does NOT inject raw CI logs into the model's feedback (that feedback is deliberately sanitized). Autonomous diagnosis from failing-step logs is left as a possible follow-up pending a prompt-injection review.Reviewer Test Plan
How to verify
The change is to a GitHub Actions workflow (
.github/workflows/qwen-autofix.yml) and the agent prompt (.qwen/skills/autofix/SKILL.md), verified by the workflow's own unit test plus replaying the affected shell/jq logic under bash against live data.npx vitest run scripts/tests/qwen-autofix-workflow.test.jspasses 48/48, including the assertions this PR updates for the watermark, pending-check, and handoff changes, and the pre-existing guards that forbid comment triggers and raw-log/URL injection (which this PR respects).2026-07-15T18:25:04Z, the head commit (a base-sync "Update branch" merge) is2026-07-16T01:58:46Z, and the maintainer's two comments are at23:50and00:12. Old logic (EFF_WM = max(pushDate, evalMarker)) →01:58:46→ both comments buried. New logic (EFF_WM = evalMarker, PRcreatedAtas the pre-first-eval floor) →18:25:04→ both comments picked up.npm run generate:settings-schemaon a clean tree produces zero diff, so the new verify-gate check fails only on a genuinely stale artifact, never falsely.run:blocks passbash -n, and CI's ownactionlint/shellcheck/yamllint/ schema-freshness steps pass on this branch.Expected vs observed: for the motivating PR, the loop would now re-target the PR (D), be able to run the generator and be told to (C), have the stale artifact caught locally before push (A), and never go silent on failure (E). Observed: all replays above produce the expected outputs and the contract test is green.
Evidence (Before & After)
Non-UI change (CI workflow + agent prompt). No screenshots. Concrete before/after of the watermark change, replayed on the motivating PR's live data:
Tested on
Environment (optional)
The workflow runs only on
ubuntu-latestGitHub-hosted runners. Verified locally under bash:scripts/tests/qwen-autofix-workflow.test.js(48/48), YAML parse,bash -non everyrun:block,npm run generate:settings-schemaidempotency on a clean tree, and replay of the watermark math against the live motivating PR.Risk & Scope
npm run generate:settings-schemaper verify (idempotent, no diff on clean trees). The watermark change is strictly safer — a real bot fix still advances the watermark via its eval marker; only unrelated base-sync commits stop advancing it.Linked Issues
Motivated by the autofix stall on #6984 (references, does not close).
中文说明
本 PR 做了什么
强化自动修复回路(
qwen-code-dev-bot),让它能从一类目前会永久卡死的 CI 失败中恢复:改了源码但没有重新生成已提交的产物。具体触发点是改动 settings schema 源码却让settings.schema.json过期——CI 的 "Check settings schema is up-to-date" 步骤会失败,但该失败对 build/typecheck/lint/Vitest 都是隐形的,于是 bot 既跑不了修复命令、也抓不到它、还会静默失联。四处协同改动闭合这个回路:给 agent 重新生成产物的工具和指令;让确定性 verify gate 镜像 CI 的新鲜度校验,使过期产物在推送前本地就失败;"这条反馈我是否处理过" 的水位线不再因无关的基线同步合并而前移(维护者评论不会被埋);失败的运行一律留下可见的交接说明 + 标记而不是静默。卡住的 pending 检查也不会再永久阻塞 PR。为什么需要
autofix bot 在一个真实 PR 上卡死:它改了 settings schema 源码却没重新生成
settings.schema.json,CI 新鲜度校验变红,而 bot(1)即便给了它精确指令也修不了,因为npm run generate:settings-schema既不在 agent 工具白名单、也不在 verify gate 里;(2)维护者评论了修复方法后,一次手动 "Update branch" 合并把 push 水位线推过了那条评论,把它埋掉,于是之后每次扫描都报 "nothing new";(3)任何在写交接文件之前就失败的运行,既不发评论也不写 eval 标记,回路就此静默,只能人工接管。这对任何"有已提交生成产物的源码"都是复发性失败模式,值得结构性修复而非逐 PR 打补丁。范围说明:本 PR 刻意守住两个既有的、有测试的设计决定——不新增评论触发的运行路径(工作流有意只在
pull_request_review:submitted触发),也不把原始 CI 日志注入模型的反馈(该反馈刻意做了净化)。基于失败 step 日志的自主诊断留作后续,待做一次 prompt-injection 评审。评审验证方案
如何验证
改动是 GitHub Actions 工作流(
.github/workflows/qwen-autofix.yml)和 agent prompt(.qwen/skills/autofix/SKILL.md),通过工作流自身的单测 + 在 bash 下用真实数据回放受影响的 shell/jq 逻辑来验证。npx vitest run scripts/tests/qwen-autofix-workflow.test.js48/48 通过,包括本 PR 为水位线/pending/交接改动更新的断言,以及禁止评论触发、禁止原始日志/URL 注入的既有守卫(本 PR 遵守它们)。2026-07-15T18:25:04Z,head commit(基线同步 "Update branch" 合并)2026-07-16T01:58:46Z,维护者两条评论在23:50和00:12。旧逻辑(max(push, eval))→01:58:46→ 两条被埋;新逻辑(EFF_WM = eval,首评前地板用 PRcreatedAt)→18:25:04→ 两条被拾起。npm run generate:settings-schema零 diff,新检查只对真正过期的产物失败。run:块通过bash -n,CI 自身的actionlint/shellcheck/yamllint/schema 新鲜度步骤在本分支均通过。预期 vs 实测:对触发 PR,回路现在会重新盯上该 PR(D)、能跑生成命令并被要求这么做(C)、在推送前本地就抓到过期产物(A)、失败时不再静默(E)。实测:回放均产出预期结果,契约测试全绿。
证据(前后对比)
非 UI 改动(CI 工作流 + agent prompt),无截图。水位线改动在触发 PR 真实数据上的前后对比:
测试情况
环境(可选)
工作流仅在
ubuntu-latestGitHub 托管 runner 上运行。本地在 bash 下验证:scripts/tests/qwen-autofix-workflow.test.js(48/48)、YAML 解析、对每个run:块bash -n、干净树上npm run generate:settings-schema幂等性、以及针对真实触发 PR 回放水位线计算。风险与范围
npm run generate:settings-schema(幂等,干净树零 diff)。水位线改动严格更安全——真正的 bot 修复仍通过其 eval 标记推进水位线,只有无关的基线同步 commit 不再推进它。关联 Issue
由 #6984 上的 autofix 卡死触发(引用,不关闭)。