fix(ci): recover /resolve requests lost to moved heads, fork pushes, 503s and drafts - #10429
Conversation
…503s and drafts Of the 294 resolutions the /resolve agent has ever produced, 17 were thrown away because the PR head moved while it worked, ten more agent runs were spent on fork PRs whose push was then rejected for lack of maintainer edits, a 503 from the permission API denied writers silently, and 182 explicit requests on drafts were answered with a skip comment. - Report result: when the lease push is declined for a moved head, fetch the new head, redo the base merge on it, take the agent's resolution for every still-conflicting file the new commits did not touch, re-run the marker / clean-merge / base-changed-files guards, and push with a lease on the new head. Anything else gives up and reports "moved" as before. The agent is never re-run. - Prepare: refuse fork PRs with "Allow edits by maintainers" off before the agent runs, via maintainerCanModify. - authorize: retry a 5xx from the permission API up to three times before treating it as a denial; still fails closed. - Prepare: drop the draft gate for the command (the review lane keeps its own). The replay functions are exercised against a real git fixture in three cases; contract pins cover the push wiring, the fork preflight, the draft gate and the retry loop.
|
Gate re-run on request — still passes at the current head.
Moving on to code review. 🔍 中文说明应请求重新执行门禁检查——在当前 head 上仍然通过。
进入代码审查。🔍 — Qwen Code · qwen3.8-max Reviewed at |
Code reviewIndependent take before reading the diff: for "lease-declined push drops the resolution", the sane fix is to fetch the new head, re-merge the base, keep the agent's version only for still-conflicting files the new head did not touch, re-verify structurally, and push with a lease on the new head. That is what this PR does — with the right conservatism: any still-conflicting file the new head touched aborts the replay and restores the original resolution. What I verified against the full workflow at the reviewed commit, beyond the diff itself:
No blockers. Two non-blocking observations: the newline-joined scope list has the same embedded-newline-filename edge as the original 'Resolution check' guard (same technique, same documented bound, fails toward giving up), and the retry adds up to 15 s to a genuinely failing authorize — both documented in the PR. TestingThis is an unattended run — PR code is never executed here. Evidence is the PR's own CI on the reviewed commit, quoted from the API:
The ubuntu Real-scenario (tmux): 中文说明代码审查读 diff 之前的独立判断：对"推送被租约拒绝就丢掉解决结果"这个问题，合理的修法是拉取新 head、重新合并 base、只对仍冲突且新 head 未触碰的文件保留 agent 的版本、重新做结构校验、再以新 head 为租约推送。本 PR 正是这样做的——且保守得当：任何仍冲突的文件只要被新 head 改过，重放就放弃并恢复原来的解决结果。 在 diff 之外，我在被审查的 commit 上对照完整 workflow 核实了：
无阻塞问题。两个非阻塞观察：换行连接的范围列表与原 'Resolution check' 守卫有同样的"文件名含换行"边界（同一技术、同样的已记录边界、失败方向是放弃）；重试最多给真正失败的 authorize 增加 15 秒——两者 PR 中均已说明。 测试这是无人值守运行——此处从不执行 PR 代码。证据是被审查 commit 上 PR 自己的 CI，摘自 API（见上表）：
真实场景（tmux）： — Qwen Code · qwen3.8-max Reviewed at |
|
Confidence: 4/5 — clean across every stage; the single reservation is that the live moved-head path is only observable after merge, which is intrinsic to default-branch workflow changes, not a doubt about the code. This reads like someone watched the lane lose real work and went after each loss class with exactly as much machinery as it earns and no more. The replay is the right shape: it never re-runs the agent, it re-holds the replayed tree to the same structural checks as the first merge, and every uncertainty gives up into today's behaviour instead of guessing. The test suite is the strongest part — real git fixtures driving the extracted production shell, scripted- Non-blocking notes for merge: the retry adds up to 15 s to a genuinely failing authorize, and the first few real 中文说明置信度：4/5 —— 每个阶段都干净；唯一的保留是真实的 head 变动路径只能合入后观测，这是默认分支 workflow 变更的固有属性，不是对代码本身的疑虑。 这看起来像是有人亲眼看着这条链路丢失真实的工作，然后对每一类损失都用了恰好够用的机制去解决、一点不多。重放的形状是对的：从不重新运行 agent，重放后的树被按与第一次合并相同的结构检查再验一遍，任何不确定情况都放弃并回退到现有行为，而不是猜测。测试套件是最强的部分——用真实 git fixture 驱动从生产脚本逐字提取的 shell、用脚本化 合入时的非阻塞提示：重试最多给真正失败的 authorize 增加 15 秒；合入后最初几次真实 — Qwen Code · qwen3.8-max Reviewed at |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Two blocking items — details in my notes above:
BASE_REFis not defined in theReport resultstep's env, so the replay path dies onunbound variableat its first expansion — exactly when a moved head triggers it — and the result comment is never posted. The fixture tests mask this by injecting the variable themselves. One-line fix: addBASE_REF: '${{ steps.prepare.outputs.base_ref }}'to the step's env.- CI is red on the workflow size gate (176,564 bytes vs the 165,876 baseline, allowance 4,096), so vitest never ran on this commit. Update
.github/workflows/.size-baselinewith the reason, or shrink the workflow.
The rest of the review was clean — direction, bounds, guards, and tests all check out. With these two fixed this should be ready. 🙏
|
@qwen-code /takeover |
qwen-code-dev-bot
added
the
autofix/takeover
|
🤝 Takeover engaged: the autofix loop now manages this PR — it will address new review feedback and resolve base conflicts until the label is removed or the round cap is reached. Remove the 中文说明🤝 已接管：autofix 循环现在管理此 PR —— 将持续处理新的评审反馈与 base 冲突，直到移除标签或达到轮次上限。移除 |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Not explored to full depth (tool budget reached): "agent 4": none — no check was cut short..
Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/qwen-code-pr-review.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
中文说明
未探索到全部深度（达到工具调用预算）："agent 4"：none — no check was cut short.。
未检查（工具限制，非阻断）：the executable-script lint — .github/workflows/qwen-code-pr-review.yml: actionlint embedded-shell source mapping is not yet supported — not linted。
— qwen3.8-max via Qwen Code /review (v0.22.2)
|
🔀 Base updated: red check(s) [Test (ubuntu-latest, Node 22.x)] pass on current main — merged current main via update-branch; CI will re-run. 中文说明🔀 已更新 base：红色检查 [Test (ubuntu-latest, Node 22.x)] 在当前 main 上通过 —— 已通过 update-branch 合入当前 main，CI 将重新运行。 |
The moved-head replay, the fork preflight and the authorize retry grow qwen-code-pr-review.yml past the 4096-byte ratchet; the growth is the recovery logic itself, so bump the baseline.
The moved-head replay merges the base branch again, but the step's env never defined BASE_REF, so under `set -u` the first replay would have aborted the step. Add the env entry and a test that walks every run block of resolve-pr and fails on any expanded uppercase variable that neither the job env, the step env, a shell assignment nor a runner builtin defines — the fixture test injected BASE_REF itself, which is why nothing caught it.
|
@qwen-code /triage |
|
Sandboxed verification: ✅ passed — merge-ready (agent verdict) - workflow run Ran the PR in an isolated, token-free container: A/B against the base build, mock-free harness assertions, targeted gates. Advisory evidence for human reviewers — not a review, an approval, or a CI check. Scripted assertions: 102 passed · 0 failed · 102 total Flakiness gate: ✅ 1 changed test file(s) x 5 identical rounds, no divergence 中文 — 判定：✅ 通过 · 可合入（agent 判定）沙箱验证在隔离、无凭证的容器中执行了该 PR 的代码（与 base 构建 A/B 对照、无 mock harness 断言、定向门禁）。仅作为评审证据，不构成评审、批准或 CI 检查。 脚本断言：102 通过 · 0 失败 · 102 总计 抖动门：✅ 1 changed test file(s) x 5 identical rounds, no divergence Verification reportPR 10429 deep verification —
|
| Cell | Scenario | Base (HEAD^1) |
Head (HEAD^2) |
|---|---|---|---|
| A | head moves on an unrelated file (b.txt) while a.txt is the conflict |
exit 1; fork stays at new head; old comment "…the update was not pushed. Re-run /resolve." — resolution lost | exit 0; replayed merge pushed; parents = [new head, origin/main]; a.txt=agent resolution, b.txt=contributor's new change, c.txt=base merge; only a.txt,c.txt differ from new head (scope guard); comment names the replay + new sha |
| B | move edits the conflicted file (a.txt) |
exit 1, moved comment | exit 1; "Replay gave up: a.txt still conflicts and the new head changed it…"; fork untouched; runner restored to original resolution branch, clean tree |
| C | push refused 403-style, head NOT moved | — | exit 1; classified permission; no replay attempted (no -replay branch); permission comment |
| D | sibling shape: move DELETES the conflicted file (delete/modify conflict) | — | exit 1; fails closed on a.txt; moved comment; original resolution restored |
| E | no move; push succeeds first try | exit 0; resolution pushed; plain comment | identical — happy path unchanged (no regression) |
| F | moved head already resolved the conflict (clean merge on replay) | — | exit 0; clean merge replayed + pushed; parents correct; a.txt=base content |
Counts: 9 cells, 44 scripted oracles, 44 pass (base-arm failures are expected-failure assertions encoded as passes). The load-bearing flip is cell A: broken → fixed on identical fixtures, differing only in the extracted run block.
Guard fidelity: the replay's three guards were compared line-by-line against Resolution check — identical marker regex ('^(<<<<<<<|>>>>>>>) ' over diff-filter=ACMRT), identical git merge-tree --write-tree "origin/${BASE_REF}" HEAD, identical comm -23 scope-guard pipeline, differing only in the reference ($new_sha vs $HEAD_FETCH_REF).
Secondary claims
authorize retry — wire oracle under a scripted gh
authorize-harness.mjs drives the extracted Check principal write permission block with real sleep and a stub gh whose gh api follows a scripted outcome plan and logs millisecond timestamps (evidence/02-authorize-retry-head-vs-base.png):
| Cell | Plan | Head | Base |
|---|---|---|---|
| R1 | 503, 503, write |
should_review=true, 3 calls, gaps 5.01s / 10.05s, retry messages name attempts 1 and 2 of 3 | should_review=false, 1 call, 20 ms — the silent denial the PR exists to fix |
| R2 | 503 ×3 | should_review=false after exactly 3 attempts, ::error:: + summary record (fail closed) |
— |
| R3 | 404 | should_review=false, 1 call — no retry for non-transient errors | — |
| R4 | write |
true, 1 call | true, 1 call (unchanged) |
| R5 | read |
false, 1 call, summary names permission is 'read' (non-writer denial unchanged) |
— |
25/25 oracles. Note authorize is shared with the review lane — the retry applies there too by construction, fail-closed outcome unchanged.
Fork preflight + draft gate — extracted Prepare pull request branch with real jq/git fixture
preflight-harness.mjs (evidence/03-fork-preflight-draft-gate.png), 16/16 oracles:
| Cell | Shape | Head | Base |
|---|---|---|---|
| P1 | fork, edits OFF | unsupported, skip reason names the fork + the setting, exit 0, no branch → no agent run |
— |
| P2 | same repo, maintainerCanModify=false |
decision=run (the false-positive guard; a bare maintainerCanModify check would refuse every in-repo PR) |
— |
| P3 | fork, edits ON | decision=run | — |
| P4 | fork, edits OFF | — | decision=run — agent run would be spent and rejected at the push (the A/B pair) |
| P5 | draft | — | skip, "PR #1 is draft." |
| P6 | draft | decision=run — draft is no longer even fetched (isDraft absent from the --json list) |
— |
The review lane keeps its own gate: "Skipping delayed review: PR #… is draft." present at line 267 of the head workflow. unsupported is consumed by Report skipped request (line 2835), so the refusal reaches the PR as a comment.
Targeted gates
| Gate | Result |
|---|---|
| vitest, head: resolve + autofix + ci-flaky-rerun + workflow-size | 463/463 pass (resolve 37, autofix 223, flaky 8, size 195) — evidence/05-targeted-gates-vitest-head.png |
Same suites, base (HEAD^1 worktree) |
455/455 pass → delta +8 tests, 0 changed, 0 lost |
bash -n (expressions stripped) on the 3 edited run blocks, both arms |
6/6 OK |
| js-yaml parse of the full workflow, both arms | OK (extraction is the witness) |
actionlint (pinned binary; live-proven: planted needs: [nonexistent-job] → rc=1) |
PR workflow clean, rc=0 |
| Size ratchet | 176,784 B ≤ 176,564 recorded + 4,096 allowance → green (see F2) |
Reviewer Test Plan walkthrough (per step)
- vitest command → passes (see Corrections for the count mismatch).
bash -n+ js-yaml on the three edited blocks → passes, both arms.- The three claimed mutation checks → all reproduced:
--forceinstead of the lease → red (2 tests); dropping thehead_repoconjunct → red; branch nameqwen-resolve/pr-N/replay→ fixture red with "could not check out" (git refuses a ref under the existingqwen-resolve/pr-Nbranch). - "After merge: live moved-head run" → not performable here (requires the workflow on the default branch). This is the one step of the plan that cannot be executed in-container; the fixture-level wire evidence above is the substitute.
Mutation matrix (vacuity) — 8 mutants, 0 survivors
All runs in a scratch worktree at HEAD (run-mutations.sh → evidence/04-mutation-matrix-8-mutants-killed.png; full logs in mut-logs/); unmutated control 37/37 green first.
| Mutant | Killed by | Attribution |
|---|---|---|
M1 remove BASE_REF env + comment |
defines every uppercase variable… → "step 'Report result' expands undefined: BASE_REF" |
the intended assertion; proves the last commit's test is non-vacuous. Note the fixture tests stayed green under M1 (they inject BASE_REF themselves) — exactly the hole the commit message describes |
M2 revert replay wiring to if false |
wires the replay into the push path… |
intended |
M3 --force-with-lease=… → --force |
pre-existing keeps the authorization and scope guards and new wires the replay… |
lease is pinned on both sides |
M4 drop [ "$head_repo" != "$REPO" ] conjunct |
refuses fork PRs… |
intended |
M5 replay branch pr-N/replay |
both fixture tests — "could not check out" (real git D/F refusal) | matches the author's claimed derivation |
M6 delete HEAD_SHA="$new_sha" |
fixture test 1's SHA oracle (expected <new> to be <orig>) + wiring pin |
behavioral control in the same file: the fixture catches a real behavioral regression, not just text |
| M7 reword "Allow edits by maintainers" pin | refuses fork PRs… pin (pre-existing pin survived via the prose comment still containing the phrase — stated for accuracy) |
positive control |
M8 ${TOTALLY_UNDEFINED_PROBE_VAR} planted in Show run artifacts |
defines every uppercase variable… → "step 'Show run artifacts' expands undefined: …" |
the env scan is genuinely job-wide, not just Report result |
Corrections (description inaccuracies, not code defects)
- The Reviewer Test Plan says "3 files, 267 tests pass (36 in the resolve file; 7 new, 3 of them driving real git repositories)". Measured at head: 268 tests across the 3 files; the resolve file has 37 tests with 8 new (the 8th,
defines every uppercase variable…, appears to be uncounted in the headline). The "3 driving real git repositories" part is accurate. .size-baselinerecords 176,564 B while the file at HEAD is 176,784 B. The gap is exactly consistent with the final commit's 3-lineBASE_REFenv addition landing after the baseline bump (commit ordering visible in the metadata). It passes via the documented 4,096-byte allowance; the next workflow PR touching this file will pay the 220 B. See F2.
Findings
No blocking findings. Two low-severity observations:
F1 (low, cosmetic). In the clean-merge replay shape (cell F — the moved head already resolved the conflict itself), the posted comment still says "only files the new commits did not touch were taken from the agent's merge", but in that shape nothing is taken from the agent's merge (the clean merge result is pushed, correctly). The tree is right; only the wording overstates. No fix suggested — flagging for the author's awareness.
F2 (low, informational). .size-baseline lags the actual file by 220 B (see Corrections #2). Gate is green via the allowance; nothing to change unless the maintainer wants the ratchet exact.
Not covered
- Live moved-head run on the default branch (Reviewer Test Plan step 4) — requires the merged workflow; the fixture wire is the substitute. Related: the double-move race (head advances again between the replay's fetch and its second push) was not interleaved; it reduces to a standard lease decline, and cell C proves decline classification after a first push.
- Replay calibration against a real emitted artifact — no GitHub token in this sandbox, so no actually-posted "moved" comment could be fetched. The replay is therefore uncalibrated: corroborating but weaker evidence is that the base arm's cell-A comment reproduces the description's quoted "Before" text verbatim (plus the artifact sentence). Calibrating would need one real
qwen-resolve-resultcomment from a production moved run. - Per-commit attribution — the checkout is depth 2 (merge commit + two parents); the metadata lists 5 commits but only
HEAD^2is reachable, so the aggregateHEAD^1..HEADdiff was verified, not each commit. - yamllint — not installable in this container (
pip3permission-denied, nopipmodule); YAML well-formedness is covered by js-yaml parse on both arms, and actionlint (live-proven) covers workflow structure. - Organisation-owned forks (declared out of scope by the PR) — they pass the new preflight when the box is ticked and surface at the push as before; not driven.
- The loss statistics (294 resolutions / 17 moved losses / 10 fork runs / 182 draft skips, run 32054217289) — author claims, no API access to verify; the mechanisms they motivate are each verified above.
- Other jobs of
qwen-code-pr-review.ymlbeyond the draft-gate spot check, and#10428-related behavior — out of this PR's diff.
Methodology
Container: the verify lane's own node:22-bookworm sample (node v22.23.2, git 2.39.5, jq 1.6, bash). All workflow code was extracted verbatim via js-yaml (extract-steps.mjs) from the head and base YAML copies — never retyped — and driven under bash --noprofile --norc with each block's own set -euo pipefail, ${{ }} expressions substituted exactly as Actions substitutes them before bash runs. The unit under test was never stubbed: the only seams are git's own url.insteadOf rewrite of the production push URL onto local bare repos (the wire oracle: post-run fork refs, pushed commit parents/trees, exact comment bodies captured by the gh stub) and the scripted gh api/gh pr view at the API boundary (real jq, real git fixtures, real sleep for the measured 5s/10s backoff). Harness scripts, raw per-cell logs (ab-matrix-raw.log, authorize-raw.log, preflight-raw.log, mutation-raw.log, mut-logs/), and the extracted blocks live in this directory; the base arm used git worktree add tmp/base-tree HEAD^1 (removed after the A/B). Every numbered assertion in assertions.json maps to a check()/exit-code/test-result that executed: 44 A/B oracles + 25 authorize + 16 preflight + 6 bash -n + 9 mutation-matrix outcomes + 2 suite-level vitest runs = 102.
Flakiness gate log
rounds=5 files=1 skipped=0
file scripts/tests/qwen-resolve-workflow.test.js: (cd .) npx --no-install vitest run --config ./scripts/tests/vitest.config.ts ./scripts/tests/qwen-resolve-workflow.test.js
per-file results (P=pass F=fail I=infra-exit, one letter per run):
scripts/tests/qwen-resolve-workflow.test.js: PPPPP
verdict: pass
summary: 1 changed test file(s) x 5 identical rounds, no divergence
--- per-invocation detail (full copy in the artifact) ---
round 1 · scripts/tests/qwen-resolve-workflow.test.js: P (exit 0)
round 2 · scripts/tests/qwen-resolve-workflow.test.js: P (exit 0)
round 3 · scripts/tests/qwen-resolve-workflow.test.js: P (exit 0)
round 4 · scripts/tests/qwen-resolve-workflow.test.js: P (exit 0)
round 5 · scripts/tests/qwen-resolve-workflow.test.js: P (exit 0)
Evidence images
Harness scripts and raw logs are in the workflow run artifacts (7-day retention).
— Qwen Code · sandboxed verification
|
Triage re-run completed without a new review.
The stage comments above were updated with the latest result. View workflow run. 上方各阶段评论已更新为最新结果。查看工作流运行。 |
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship — CI landed green after the review. ✅
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Reviewed — no blockers. Suggestions are inline.
3 Suggestion-level finding(s) this review confirmed are already reported on this PR and are not repeated:
- R1-3 draft-gate pin names a job that does not exist (
delayed-review) — already reported (comment 3883650503) - R1-4 retry loop pinned only by string presence, never executed — already reported (comment 3883650518)
- R1-5 replay give-up / structural-refusal branches have no fixture coverage — already reported (comment 3883650522)
Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/qwen-code-pr-review.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
Deferred under the convergence posture (round 2, not a blocker) — recorded, not requested in this round:
.github/workflows/qwen-code-pr-review.yml:383 — [review] retry gate's hand-enumerated transient wordings miss Go client-side failures.github/workflows/qwen-code-pr-review.yml:2568 — [review] fork-pushability preflight also refuses dry_run dispatches, which never push.github/workflows/qwen-code-pr-review.yml:3002 — [review] replay's merge-tree re-check passes by construction; the real base invariant is unstated.github/workflows/qwen-code-pr-review.yml:3007 — [review] replay duplicates the four structural checks of 'Resolution check'scripts/tests/qwen-resolve-workflow.test.js:1158 — [review] fork-guard test never pins the --json field list it reads from
中文说明
已审查——无阻断问题。 建议见行内评论。
本轮确认的 3 条建议级发现已在 PR 上报告过，不再重复发布（列表见上方英文部分）。
未检查（工具限制，非阻断）：the executable-script lint — .github/workflows/qwen-code-pr-review.yml: actionlint embedded-shell source mapping is not yet supported — not linted。
收敛姿态下延后（第 2 轮，非阻断）——已记录，本轮不要求修改：共 5 条（原文未翻译，列表见上方英文部分）。
— qwen3.8-max via Qwen Code /review (v0.22.3)
The post-agent scrub landing in #10428 removes the workspace .git/config — and with it the `origin` remote — before any credentialed step runs, so the replay's `git fetch origin` would fail there. Fetch by URL, with an env hook (empty in production) that the fixture test points at its bare repository; a pin forbids `git fetch origin` in the replay.
|
@qwen-code /triage |
|
Sandboxed verification: ❌ not passed — findings reported (agent verdict) - workflow run Ran the PR in an isolated, token-free container: A/B against the base build, mock-free harness assertions, targeted gates. Advisory evidence for human reviewers — not a review, an approval, or a CI check. Scripted assertions: 124 passed · 0 failed · 124 total Flakiness gate: ✅ 1 changed test file(s) x 5 identical rounds, no divergence 中文 — 判定：❌ 不通过 · 报告了发现（agent 判定）沙箱验证在隔离、无凭证的容器中执行了该 PR 的代码（与 base 构建 A/B 对照、无 mock harness 断言、定向门禁）。仅作为评审证据，不构成评审、批准或 CI 检查。 脚本断言：124 通过 · 0 失败 · 124 总计 抖动门：✅ 1 changed test file(s) x 5 identical rounds, no divergence Verification reportPR 10429 deep verification (follow-up round) —
|
| # | Finding | Severity | Status at new head |
|---|---|---|---|
| F1 | Clean-merge replay comment overstates ("only files the new commits did not touch were taken from the agent's merge" — in that shape nothing is taken from it) | low | stands — re-measured in cell F1 (A/B table): wording unchanged, pushed tree correct (a.txt = main content). Cosmetic only |
| F2 | .size-baseline lags the actual workflow size |
low | stands, widened — gap 220 B → 744 B (recorded 176,564 vs measured 177,308). Gate still green via the 4,096-byte allowance, live-proven in both directions (see Targeted gates) |
| Corr-1 | Description test-count off-by-one ("267 / 36 / 7 new") | nit | stands — the delta added no new it() blocks; measured remains 268 tests across the 3 files, 37 in the resolve file, 8 new (29 → 37 vs base) |
Scope selection
This round's central claim (the delta): the replay fetches the moved head by URL (${RESOLVE_ORIGIN_URL:-https://github.com/${REPO}.git}), so it works after the post-agent scrub removes .git/config/the origin remote; the previous round's git fetch origin dies in that state. RESOLVE_ORIGIN_URL is '' in production → default URL https://github.com/QwenLM/qwen-code.git (job env REPO = ${{ github.repository }}, the base repo — where refs/pull/N/head is mirrored; verified correct).
Carried claims re-verified: the replay itself vs base (previous round's central claim, re-established on the moved base), authorize retry, fork preflight, draft gate.
Central claim — A/B table (delta: fetch by URL vs fetch by remote)
Harness: ab-fetch-harness.mjs drives the verbatim js-yaml-extracted Report result block from three workflow versions — base = HEAD^1 (no replay), head = HEAD (fetch by URL), control = 3a00591246 (previous head, git fetch origin; the exact reverse of the delta hunk) — under bash --noprofile --norc with the block's own set -euo pipefail, ${{ }} substituted exactly as Actions does. Real git wire: bare origin.git (main + refs/pull/1/head), bare fork.git (refs/heads/feature, push target), runner checkout holding the agent's resolution. Production URLs reached via url.insteadOf injected through GIT_CONFIG_* env vars (works with .git/config deleted — required for the scrub cells); only the comment API is simulated (gh stub captures the exact body). Scrub states: none; remote ([remote "origin"] removed, identity kept); full (.git/config deleted, identity via env); full-noid (deleted, identity nowhere). Witness: evidence/01-ab-fetch-matrix-head-vs-control.png; raw per-cell logs in raw/<cell>/.
| Cell | Arm · scrub · scenario | Result |
|---|---|---|
| A1 | base · none · unrelated move | exit 1; old "Re-run /resolve." comment; fork stays at new head — resolution lost (carried-over broken cell re-established on the moved base) |
| A2 | head · none · unrelated move | exit 0; replay pushed — no regression while origin still exists |
| A3 | head · remote-scrub · unrelated move · production URL shape (hook empty) | exit 0; fetch-by-URL needs no remote; pushed tree keeps agent a.txt + contributor b.txt + base c.txt; parents [new head, origin/main]; scope guard a.txt,c.txt only; agent commit message reused |
| A3h | head · remote-scrub · hook → fixture origin path | exit 0 — hook form parity |
| A4 | control · remote-scrub · unrelated move | exit 1; Replay gave up: could not fetch the new head.; old moved comment; fork untouched; runner restored, clean tree — the load-bearing flip vs A3 |
| A5 | control · none · unrelated move | exit 0 — the control is viable while the remote exists; the flip is attributable to the scrub state, not a phantom bug |
| A6 | head · full-scrub · unrelated move | exit 0 — replay survives outright .git/config deletion (local refs survive in packed-refs; identity via env) |
| A7 | head · full-scrub-noid · unrelated move | exit 1, fail-closed: give-up, moved comment, fork untouched, runner restored — see F3 |
| B1 | head · remote-scrub · move edits conflicted file | exit 1; give-up names a.txt; restored (carried-over shape holds under scrub) |
| E1 | head · remote-scrub · push declined, head NOT moved | exit 1; "declined for another reason"; restored |
| F1 | head · remote-scrub · moved head already resolved the conflict | exit 0; clean merge replayed + pushed; F1 wording re-measured |
| C1 | head · remote-scrub · permission-style refusal (rejecting pre-receive hook, 403-shaped) | exit 1; classified permission; no replay attempted (moved ref never fetched, no -replay branch) |
Counts: 12 cells, 59 scripted oracles, 59 pass (expected base/give-up failures are encoded as passing assertions). Guard fidelity and the rest of the replay were verified in the previous round and are byte-unchanged outside the fetch line + env entry (delta diff verified).
Secondary claims (re-measured at new head)
secondary-harness.mjs, witness evidence/02-authorize-preflight-head-vs-base.png, 27/27 oracles:
- authorize retry (extracted head + base blocks, scripted
gh api, realsleep): R1503,503,write→ headshould_review=truein 3 calls, gaps 5.01s / 10.01s, retry messages name attempts 1–2 of 3; base: first 503 = silent denial in 1 call (the flip is preserved). R2503×3→ fail closed after exactly 3,::error::+ summary. R3404→ denial in 1 call (no retry for non-transient). R4write→ both arms allow in 1 call. R5read→ deny, summary namespermission is 'read'. - fork preflight + draft gate (extracted
Prepare pull request branch, real jq + real git fixture): fork + edits OFF →unsupported, reason names fork + setting, no branch → no agent run (head) vs base runs it (the A/B pair); same-repo withmaintainerCanModify=falsestill runs (false-positive guard holds); fork + edits ON runs; base refuses drafts ("PR pre-release: fix ci #1 is draft."), head runs them and no longer even fetchesisDraft.
Mutation matrix — 10 mutants, 0 survivors
Scratch worktree at HEAD; unmutated control 37/37 green first. Witness evidence/03-mutation-matrix-10-mutants-killed.png; full vitest logs raw/mut-*.log.
| Mutant | Killed by | Note |
|---|---|---|
M0a revert delta: URL fetch → git fetch origin |
wires the replay into the push path… (the two new pins) |
fixture tests stayed green — the fixture checkout has an origin remote, so git fetch origin succeeds there; same masking pattern as M1. The behavioral difference is covered by A/B cells A3/A4 instead |
M0b remove RESOLVE_ORIGIN_URL step env |
defines every uppercase variable… |
fixture injects its own — masked there too; the env scan catches it |
M1 remove BASE_REF step env |
defines every uppercase variable… |
carried-over, re-confirmed at new head |
M2 wiring → if false |
wires the replay… |
|
M3 lease → --force |
keeps the authorization and scope guards… and wires the replay… |
lease pinned on both sides |
M4 drop [ "$head_repo" != "$REPO" ] conjunct |
refuses fork PRs… |
|
M5 replay branch pr-N/replay |
both fixture tests ("could not check out" — real git D/F refusal) | |
M6 delete HEAD_SHA="$new_sha" |
fixture SHA oracle + wires the replay… |
behavioral control in the same file |
| M7 reword Allow-edits prose | refuses fork PRs… |
positive control |
M8 plant undefined var in Show run artifacts |
defines every uppercase variable… |
env scan is job-wide |
Reviewer Test Plan walkthrough (per step)
- vitest command → re-run at new head: 4 files green (see Targeted gates); counts differ from the description as in Corr-1.
bash -n+ js-yaml → re-run both arms + control: 9/9 blocks OK, 3/3 YAMLs parse (extraction is the witness).- The three claimed mutation checks → all reproduced in the matrix:
--force(M3, red in 2 tests), dropping thehead_repoconjunct (M4, red), branchpr-N/replay(M5, fixture red). - "After merge: live moved-head run" → still not performable in-container (needs the workflow on the default branch). The fixture-level wire evidence is the substitute.
Targeted gates
| Gate | Result |
|---|---|
| vitest head: resolve + autofix + ci-flaky-rerun + workflow-size | 463/463 pass — witness evidence/04-targeted-gates-vitest-head.png |
Same suites, base (HEAD^1) |
455/455 pass → delta +8 tests, 0 changed, 0 lost (resolve file 29 → 37) |
qwen-pr-review-workflow.test.js (the wider suite for this file, not run in the previous round) |
179/179 both arms — no regression |
bash -n (expressions stripped) on all 3 edited blocks × 3 arms |
9/9 OK |
| js-yaml parse of head, base, control workflows | 3/3 OK |
actionlint (pinned binary; live-proven: planted needs: [nonexistent-job-xyz] → rc=1) |
PR workflow clean, rc=0 |
Size ratchet (check-workflow-size.sh, WORKFLOW_SIZE_BASE_SHA=HEAD^1) |
green: 177,308 ≤ 176,564 + 4,096. Live-proven both directions: tightening the allowance to 500 flags exactly "grew to 177308 bytes, 744 over its recorded 176564" (see F2) |
Findings
F3 (medium, conditional) — the scrub state the delta cites also kills the replay's git identity; under it the replay gives up with a misleading reason before committing. The delta's commit message says the post-agent scrub "removes the workspace .git/config (and with it the origin remote)". The replay's merge/commit needs a committer identity, and that identity is the user.name/user.email the Prepare step writes into the same .git/config — so a scrub that removes the file (or one shaped like this repo's existing resanitize-git-config.sh, whose local allowlist keeps remote.*.url but unsets user.*) leaves the replay identity-less. Measured chain (cell A7, reproducing command: node tmp/pr10429-verify-20260829-034431/ab-fetch-harness.mjs): with .git/config gone and no identity anywhere, git merge --no-commit itself exits 128 "Committer identity unknown" before any conflict exists; the replay swallows that stderr by design, finds nothing unmerged, and reports Replay gave up: merging main into <sha> changes nothing. — misleading, since the merge never ran. The outcome is still fail-closed (exit 1, moved comment, fork untouched, runner restored — all asserted), and pre-scrub production (today) is unaffected; but once #10428's scrub lands, the replay the delta exists to save would die in exactly the state the delta targets. The PR's own fixture masks this: runReplay injects GIT_AUTHOR_NAME/GIT_COMMITTER_NAME env. This is conditional because #10428's scrub is not in this tree (see Not covered) and a runner-global fallback identity cannot be measured from here.
Measured fix probe (scratch, not part of the PR)
Restoring the bot identity at the top of replay_on_moved_head (the same two git config lines Prepare uses) converts the dead state into a working replay with zero collateral: fix-probe.mjs against scratch block extracted/report-result.headfix.sh, witness evidence/05-f3-identity-fix-probe.png — A7-fix: exit 0, replayed commit pushed on the new head with the agent's resolution kept; A3-fix (benign production shape): byte-equivalent tree and scope-guard outcome. 7/7 assertions. Note the PR's fixture suite cannot pin this axis either way — it supplies GIT_AUTHOR/COMMITTER env, the same masking as M0a — so if the patch lands it should ship with a fixture that runs the replay with .git/config removed and no identity env.
F1 (low, cosmetic, carried over — stands). Cell F1: the clean-merge replay pushes the correct tree while the comment still claims files "were taken from the agent's merge"; in that shape nothing is taken from it.
F2 (low, informational, carried over — widened). .size-baseline records 176,564 B; the file at the new head is 177,308 B — gap now 744 B (was 220), grown by this round's delta (+9/−1 workflow lines). Green via the documented 4,096 allowance; the next workflow PR touching this file pays it unless the ratchet is bumped in the same PR.
Not covered
- Live moved-head run on the default branch (Reviewer Test Plan step 4) — requires the merged workflow. The double-move race (head advances again between the replay's fetch and its second push) was not interleaved; it reduces to a second lease decline →
movedcomment, fail-closed. - fix(ci): run /resolve without the container sandbox and pin its CLI version #10428's scrub itself — not in this tree; F3 is conditional on what it removes. Closest available evidence inspected:
.github/scripts/resanitize-git-config.sh(the autofix lane's scrub), whose local allowlist keepsremote.*.url|fetch|pushurlbut unsetsuser.*— i.e. under that shape theoriginremote survives and the identity does not, the mirror image of the delta's assumption. Worth reconciling with fix(ci): run /resolve without the container sandbox and pin its CLI version #10428's author. - Replay calibration against a real emitted artifact — no GitHub token in this sandbox; the replay remains uncalibrated. Corroboration: the base arm's cell-A1 comment reproduces the description's quoted "Before" text verbatim.
- Per-commit exercise of commits 1–4 — all six commit objects are present but the graph is grafted (depth 2); the aggregate
HEAD^1..HEADdiff was verified, the previous round verified through commit 5, and commit 6 (9ab05c92f9) is individually attributed (recorded parent3a00591246; tree diff matches its message; the A/B targets exactly it). - yamllint — still not installable (
pip3: Permission denied); YAML well-formedness covered by js-yaml parse ×3 and live-proven actionlint. - Organisation-owned forks (declared out of scope by the PR), the loss statistics (author claims, no API), and the
Show run artifacts/Upload run artifacts/Report skipped requeststeps beyond the env-scan probe.
Methodology
Container: the verify lane's own node:22-bookworm sample (node v22.23.2, git 2.39.5, jq, bash). All workflow code was extracted verbatim via js-yaml (extract-steps.mjs) from three YAML versions — head (HEAD), base (HEAD^1), delta-control (3a00591246) — never retyped, and driven under bash --noprofile --norc with each block's own set -euo pipefail, ${{ }} expressions substituted exactly as Actions substitutes them before bash runs. The unit under test was never stubbed: git seams are url.insteadOf rewrites of the two production URLs onto local bare repos, injected via GIT_CONFIG_COUNT/KEY/VALUE env vars so they apply even with .git/config deleted (the state under test); the API seams are a scripted gh (authorize plan + millisecond timestamps, real sleep) and a comment-capturing gh stub. Scrub states were applied to the runner checkout as: git config --local --remove-section remote.origin (remote only) and rm .git/config (full), with precondition assertions that the remote is gone while refs/remotes/origin/main survives. Harnesses (ab-fetch-harness.mjs, secondary-harness.mjs, run-mutations.mjs, fix-probe.mjs) and raw per-cell logs (raw/) live in the artifact dir; the base and mutation worktrees (tmp/base-tree, tmp/mut-tree, sharing the root node_modules — the PR touches no dependencies) were removed after use. Every number in assertions.json maps to an executed check: 59 A/B + 27 secondary + 11 mutation-matrix outcomes (control + 10 kills) + 7 fix probe + 9 bash -n + 3 YAML parses + 2 actionlint + 2 size-ratchet + 4 vitest gate runs = 124.
Flakiness gate log
rounds=5 files=1 skipped=0
file scripts/tests/qwen-resolve-workflow.test.js: (cd .) npx --no-install vitest run --config ./scripts/tests/vitest.config.ts ./scripts/tests/qwen-resolve-workflow.test.js
per-file results (P=pass F=fail I=infra-exit, one letter per run):
scripts/tests/qwen-resolve-workflow.test.js: PPPPP
verdict: pass
summary: 1 changed test file(s) x 5 identical rounds, no divergence
--- per-invocation detail (full copy in the artifact) ---
round 1 · scripts/tests/qwen-resolve-workflow.test.js: P (exit 0)
round 2 · scripts/tests/qwen-resolve-workflow.test.js: P (exit 0)
round 3 · scripts/tests/qwen-resolve-workflow.test.js: P (exit 0)
round 4 · scripts/tests/qwen-resolve-workflow.test.js: P (exit 0)
round 5 · scripts/tests/qwen-resolve-workflow.test.js: P (exit 0)
Evidence images
Harness scripts and raw logs are in the workflow run artifacts (7-day retention).
— Qwen Code · sandboxed verification
…replay Address the second-round review findings on the /resolve recovery paths: - classify_push_failure now tests the lease-decline signature (stale info, force-with-lease, non-fast-forward, fetch first) BEFORE the permission patterns: git echoes the destination branch into the push log, so a moved head on a branch named e.g. fix/permission-prompt or fix-403-error matched the permission arm first and the replay never ran — the exact loss class this PR recovers. - The replay handles resolutions that deleted the conflicted file (delete/modify conflicts): checkout fails on a path absent from the resolved commit, so take the deletion via git rm instead. - The replay-success comment no longer claims files were taken from the agent's merge when the replay merged clean, and says that the run artifact describes the original resolution, not the pushed tree. - Tests: behavioural coverage for the authorize retry loop against a scripted gh (3 attempts, 5s/10s backoff, fail-closed denial arms) and for the classifier against real push logs (branch-name echoes, re-classification after a failed replay push); fixture coverage for the empty-merge give-up and the deletion resolution; the draft-gate pin now names the real delay-automatic-review job without the silent whole-file fallback.
|
🔀 Base updated: red check(s) [Test (ubuntu-latest, Node 22.x)] pass on current main — merged current main via update-branch; CI will re-run. 中文说明🔀 已更新 base：红色检查 [Test (ubuntu-latest, Node 22.x)] 在当前 main 上通过 —— 已通过 update-branch 合入当前 main，CI 将重新运行。 |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Reviewed.
7 Suggestion-level finding(s) this review confirmed are already reported on this PR and are not repeated:
- R13-1 size ratchet entry under-records the workflow by one byte (184442 vs measured 184443) — already reported (round-8 deferred list R8-1 for the same entry; triage stage-3 confidence comment names the one-byte baseline nit still open)
- R13-2 replay hand-copies Resolution check's structural checks with no equivalence pin — already reported (round-2 deferred list yml:3007; round-5 deferred list; R6-3; R11-2)
- R13-3 fork-refusal test does not pin the maintainerCanModify --json field list or jq site — already reported (round-2 deferred list test.js:1158; rounds 5 and 9; R11-7)
- R13-4 authorize-retry alternates beyond the 503 shape have no behavioural pin — already reported (R6-6; rounds 3-5 deferred lists; R11-4)
- R13-5 replay scope-guard and marker-guard rejection branches have no fixture on any lane — already reported (R6-2; R1-5, comment 3883650522; round-5 deferred list yml:3004; R11-9)
- R13-8 retry classifier omits rate-limit (HTTP 429 / 403 rate limit) wordings — already reported (round-7 deferred list R7-1; round-9 deferred Critical restating it; R11-3)
- R13-9 replay merge-tree re-check is structurally dead — already reported (round-2 deferred list yml:3002; round-5 deferred list; R6-3)
Not explored to full depth (tool budget reached): "agent 3c": none — all checks I started reached a conclusion..
Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/qwen-code-pr-review.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
中文说明
已审查。
本轮确认的 7 条建议级发现已在 PR 上报告过，不再重复发布（列表见上方英文部分）。
未探索到全部深度（达到工具调用预算）："agent 3c"：none — all checks I started reached a conclusion.。
未检查（工具限制，非阻断）：the executable-script lint — .github/workflows/qwen-code-pr-review.yml: actionlint embedded-shell source mapping is not yet supported — not linted。
— qwen3.8-max via Qwen Code /review (v0.22.3)
|
🤖 Reviewed the latest feedback — no changes needed. Why, point by point: · 已审阅最新反馈——无需改动。逐点说明原因如下： Autofix round: no changes neededI triaged every feedback point newer than the last evaluation. Nothing this round requires a code change:
Failed checks — triaged as not caused by this PRThe four failing jobs are all defined in
Local verification evidence
中文说明Autofix 轮次：无需改动我逐一分诊了上次评估之后的所有反馈。本轮没有任何需要修改代码的事项：
失败检查项——分诊结论：与本 PR 无关四个失败的 job 全部定义在
本地验证证据
Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。 🧠 Handled by Qwen Code · model/模型 |
|
🤖 Reviewed the latest feedback — no changes needed. Why, point by point: · 已审阅最新反馈——无需改动。逐点说明原因如下： Autofix review round: no action neededThis round's feedback contained no reviews, no inline comments, and no issue-level comments — only two failed CI checks. After evidence-based triage, neither represents a code defect that can or should be addressed by a change in this PR, so no commit was made. Triage of the failed checks1. 2. Local verification (evidence the tree is healthy)Every trusted runnable check passes on the current HEAD, confirming the cancellation hid no real failure:
No code changes were made this round; the branch stays at 中文说明Autofix 审查轮次：无需处理本轮反馈中没有 review、没有行内评论、也没有 issue 级评论——只有两个失败的 CI 检查。经过基于证据的分类，二者均不代表本 PR 能够或应当通过代码变更解决的缺陷，因此本轮没有提交任何变更。 失败检查的分类1. 2. 本地验证（证明分支状态健康的证据）当前 HEAD 上所有可信的可运行检查均通过，确认该取消并未掩盖真实失败：
本轮没有做任何代码变更；分支保持在 Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。 🧠 Handled by Qwen Code · model/模型 |
|
@qwen-code /triage |
|
Sandboxed verification: ❌ not passed — findings reported (agent verdict) - workflow run Ran the PR in an isolated, token-free container: A/B against the base build, mock-free harness assertions, targeted gates. Advisory evidence for human reviewers — not a review, an approval, or a CI check. Scripted assertions: 214 passed · 2 failed · 216 total Flakiness gate: ✅ 1 changed test file(s) x 5 identical rounds, no divergence 中文 — 判定：❌ 不通过 · 报告了发现（agent 判定）沙箱验证在隔离、无凭证的容器中执行了该 PR 的代码（与 base 构建 A/B 对照、无 mock harness 断言、定向门禁）。仅作为评审证据，不构成评审、批准或 CI 检查。 脚本断言：214 通过 · 2 失败 · 216 总计 抖动门：✅ 1 changed test file(s) x 5 identical rounds, no divergence Verification reportPR 10429 deep verification (fourth follow-up round) —
|
| # | Finding | Severity | Status at new head |
|---|---|---|---|
| F2 | .size-baseline lags the actual workflow size |
low | fixed at the 1,321-byte scale; re-drifted to 1 byte. The re-record commit landed exactly as prescribed (184,442, matching the file as then measured); four subsequent merges of main brought in 1 byte. Gap re-measured: recorded 184,442 vs actual 184,443. Attribution: main itself carries the same unrecorded +1 byte at the base tip (173,140 vs 173,139, both frozen at HEAD^1). See Findings — severity downgraded to nit |
| F3 | Under the scrub state, the replay died identity-less | medium | fixed — holds. Re-measured cell H1 (full-scrub-no-id): exit 0, replay pushed, committer qwen-code-dev-bot <qwen-code-dev-bot@users.noreply.github.com> (see 01-ab-pushed-base-vs-head.png) |
| F1 | Clean-merge replay comment overstates | low | fixed — holds. Re-measured cell H5: conditional clause present in the archived comment (raw/h5-comment.txt); pushed.diff proves nothing came from the agent merge |
| Corr-1 | Description test counts off | nit | stands. Re-measured at new head: description says "3 files, 267 tests (36 in resolve; 7 new)"; actual 3 files 285 (resolve 48, autofix 229, ci-flaky 8), resolve grew 29 → 48 = 19 new vs base |
| M-PAREN | Parenthesised-lease guard unpinned | nit | fixed — holds. Mutant M-BAREWORD (parenthesised+anchored → bare words) killed by classifies a moved head by its lease decline… |
| L13 | Branch named x(non-fast-forward) + genuine 403 classified moved |
nit | fixed — holds. Three-arm matrix on real hook-decline logs: C6/C7 head permission, intermediate control (prev round's unanchored shape) moved, base permission |
Scope selection
This round's delta (commit 8b01a30f): the size-baseline re-record closing round-3's F2-recurrence. Verified as the central claim below (ratchet cells in three directions + attribution of the residual byte).
Carried claims, all re-measured because the input closure changed (new base, workflow touched by merges of main): (1) the moved-head replay + pushed-tree preservation in Report result — the PR's central behavioural claim; (2) the classifier's moved-before-permission ordering with parenthesised, end-of-line-anchored reasons; (3) literal pathspecs in the replay's per-file git commands; (4) authorize retry, fork preflight + same-repo conjunction, draft-gate flip, deletion replay, clean-merge give-up shapes. Explicitly out of scope: the live moved-head run on the default branch, the loss statistics in the description (author claims, no API), org-owned forks (declared out of scope by the PR).
Central claim of this round — the size re-record (F2 status)
Measured at the verified head (gates-size.mjs; witness evidence/06-size-ratchet-cells.png; logs raw/size-gate-*.log):
| Cell | Result |
|---|---|
actual file size (git cat-file -s) |
184,443 |
| recorded baseline entry | 184,442 — gap 1 byte |
gate as CI wires it (.github/scripts/check-workflow-size.sh, no WORKFLOW_SIZE_BASE_SHA; ci.yml:422) |
green, zero diagnostics for the file (gap ≤ 4,096 allowance; the gate is silent under the allowance by design — no branch reports small overruns) |
| failure direction (scratch tree with the BASE's entry, 173,139) | exit 1, live: ::error … grew to 184443 bytes, 11304 over its recorded 173139 |
| fix cell (entry := 184,443) | green, zero diagnostics for the file |
| base tip itself | file 173,140 vs its own recorded 173,139 — main carries the identical +1-byte drift at HEAD^1 |
PR's delta to .size-baseline |
exactly one line: -173139 / +184442 qwen-code-pr-review.yml |
Attribution: the re-record was accurate when made (previous round measured the file at 184,442 at dc5a24b5; the commit recorded 184,442). The residual byte arrived with one of the four merges of main that followed, and matches main's own unrecorded +1 byte frozen at the base tip. Precise commit-level attribution is impossible at depth 2 (intermediate objects unreachable); the fix is the same either way — see Findings.
Carried central claim — replay + pushed-tree A/B (re-measured)
The entire Report result run block, js-yaml-extracted from head and base and ${{ }}-substituted exactly as Actions does, driven under bash --noprofile --norc with its own set -euo pipefail against a two-bare-repo wire (origin mirror holding refs/pull/1/head + fork push target reached through the production-shaped x-access-token URL via url.insteadOf env rewrites), a gh stub capturing the comment body, and the full scrub state (.git/config deleted, identity nowhere). Harness ab-pushed.mjs; witness evidence/01-ab-pushed-base-vs-head.png; per-cell logs raw/.
| Cell | Arm · scrub · scenario | Result |
|---|---|---|
| B1 | base · full-scrub · unrelated move | exit 1 — calibration cell. The "moved" comment byte-matches the description's quoted "Before" sentence; fork untouched (still the moved head); no pushed/ record; no replay attempted; runner left on the original resolution |
| H1 | head · full-scrub · unrelated move | exit 0 — the claim. Fork branch = replay commit (parents [moved head, origin/main]); committer = bot identity; author + message reused from the agent commit; tree = agent a.txt + contributor b.txt + base c.txt; pushed.sha == pushed commit; pushed.diff byte-equals an independent recompute of origin/main...HEAD and carries the agent resolution; comment names the moved head, points at qwen-resolve-pr-1-pushed, carries the conditional clause and the original-artifact clarification; no token in stdout/stderr/comment |
| H2 | head · full-scrub · head did NOT move | exit 0 — direct push of the original resolution; no pushed/ record; plain comment |
| H3 | head · full-scrub · move edits the conflicted file | exit 1 — give-up names the file and the original head; fork NOT rewritten; no pushed/ record; moved comment; runner restored clean on the original resolution |
| H4 | head · full-scrub · unrelated move · fork hook declines the replay push | exit 1 — the stale first push never reached the hook (1 invocation total: stale lease declines client-side before pack transfer, measured); re-classified from the second log (reason=permission); permission comment; fork untouched; no pushed/ record |
| H5 | head · full-scrub · clean-merge replay (move adopts base side) | exit 0 — replay merged clean, took nothing from the agent's merge (pushed.diff proves it); conditional clause present (F1 holds) |
| H6 | head · no scrub · unrelated move | exit 0 — no regression while config is intact |
Counts: 7 cells, 51 scripted oracles, 51 pass (expected base failure encoded as a passing assertion).
Secondary claim — classifier three-arm matrix on real logs
ab-classifier.mjs extracts classify_push_failure from head and base, plus the named intermediate control (the previous round's classifier: moved-first + parenthesised, anchor removed — a one-character surgical revert). Input bytes are real git push output — real stale --force-with-lease declines on branches named fix/permission-prompt / fix-403-error / fix/non-fast-forward-retry, a real non-FF (fetch first), real silent pre-receive-hook declines on branches named x(non-fast-forward) / x(non-fast-forward)y / fix/non-fast-forward-retry. The refusing-to-allow phrase (C9) is hook-produced — real transport bytes through the real protocol; only the HTTP-403 fatal line is synthetic, since no HTTP status exists locally — labeled where used. Witness evidence/02-classifier-three-arm-real-logs.png; logs archived at raw/real-push-logs.txt.
| Check | head | intermediate (prev round) | base | Note |
|---|---|---|---|---|
C1 stale · feature |
moved | moved | moved | real |
C2 stale · fix/permission-prompt |
moved | moved | permission | base loses the request |
C3 stale · fix-403-error |
moved | moved | permission | base loses the request |
C4 stale · fix/non-fast-forward-retry |
moved | moved | moved | base saved by bare-word echo |
C5 real (fetch first) |
moved | moved | moved | real |
C6 hook · x(non-fast-forward) + 403 |
permission | moved | permission | the L13 shape |
C7 hook · x(non-fast-forward)y + 403 |
permission | moved | permission | mid-name parens |
C8 hook · fix/non-fast-forward-retry + 403 |
permission | permission | permission | bare-word echo, no parens |
C9 hook phrase refusing to allow … workflow |
workflow_scope | workflow_scope | workflow_scope | priority preserved |
| C10 pure 403 / C11 unknown / C12 scope-beats-moved | permission / other / workflow_scope | same | same | |
| C13 re-classification on head | first=moved → second=permission | — | — | real first log, 403 second |
41/41 pass. (stale info)/(fetch first) can never appear verbatim in a branch name (spaces are illegal in refnames), so (non-fast-forward) is the only legally reachable anchor edge — C6/C7 cover exactly it.
Secondary claim — literal pathspecs, head vs one-line-revert control
ab-literal.mjs drives the full extracted block with a conflicted file named a[1].txt beside its wildcard sibling a1.txt; control = head with only export GIT_LITERAL_PATHSPECS=1 removed. Witness evidence/03-literal-pathspec-ab.png.
| Cell | Scenario | head | control |
|---|---|---|---|
| G1/G1c | move touches ONLY the sibling | exit 0 — agent resolution + sibling edit both intact | exit 1 — spurious give-up, wrong reason (blames the untouched conflicted file); request lost |
| G2/G2c | move edits the conflicted file | exit 1 — correct reason | exit 1 — correct outcome (union match still sees the change) |
| G3/G3c | deletion resolution + sibling move | exit 0 — file stays deleted, sibling alive | exit 1 — gives up |
18/18 pass. Blast radius bounded by construction: every control give-up fired before any commit/push — lost requests, never wrong trees.
Carried claims — authorize retry + prepare preflights (re-measured)
ab-secondary.mjs — witness evidence/04-carried-claims-secondary.png. 33/33 pass.
- Authorize retry (scripted
gh, sleep stubbed): head: 503,503,write → allowed, 3 calls, backoff 5s/10s; 503×3 → denied fail-closed after exactly 3; 404 → denied on the first call, no sleep; HTTP 500 and connection-reset both retried; non-writer denied. Base arm flip: first 503 denies immediately, 1 call — the loss this PR fixes. - Prepare preflights (whole extracted block, real origin): fork PR with edits off →
unsupportedbefore any agent work (context.md never written, resolution branch untouched, reason names the setting); same-repo PR withmaintainerCanModify=falseis NOT refused (conjunction holds, decision=run); draft same-repo PR →decision=runat head while base still refuses it ("is draft") — the flip; fork with edits on → runs; closed PR → skip; deleted head repo → unsupported; head'sgh pr viewasks formaintainerCanModifyand no longer asks forisDraft, base's still asks forisDraft. Base lets the unpushable fork run (decision=run) — the wasted agent run this PR eliminates.
Mutation matrix — 11 mutants, 11 killed, 0 survivors
Scratch worktree at HEAD; unmutated control 48/48 green first; each mutant is ONE surgical YAML edit (occurrence-count asserted = 1), oracle the resolve suite. Witness evidence/05-mutation-matrix-11-killed.png; per-mutant logs raw/mutations/.
| Mutant | Outcome | Killed by (intended assertion) |
|---|---|---|
M-PUSHED-REC remove the pushed/ recording lines |
killed | wiring pin wires the replay into the push path… |
M-PUSHED-POS move Upload pushed tree before Report result |
killed | wiring pin (position) |
M-PUSHED-GATE drop always() from its gate |
killed | wiring pin (gate string) |
M-PUSHED-NAME drop the -pushed suffix |
killed | wiring pin (name) |
M-LITERAL remove export GIT_LITERAL_PATHSPECS=1 |
killed | both glob-sibling fixture tests, behavioural — fail with the exact spurious give-up measured in G1c |
M-ANCHOR drop the $ anchor |
killed | branch-echo classifyLog case |
| M-BAREWORD parenthesised+anchored → bare words | killed | same classifyLog case (M-PAREN gap stays pinned) |
| M-COMMENT-PUSHED remove the pushed-artifact sentences | killed | wiring pin + comment pin (2 tests) |
| M-SLEEP backoff 5s → 7s (positive control) | killed | the retry tests (same file) |
M-FORCE lease → --force (PR's Test-Plan mutant #1) |
killed | wires the replay… + the scope-guard pin (2 tests) |
M-FORKCHECK drop the $head_repo != $REPO half (Test-Plan mutant #2) |
killed | refuses fork PRs the bot cannot push to before spending an agent run |
The pushed-recording pins are text-level by construction; the E2E cell H1 supplies the behavioural proof that the pinned lines execute and produce the right bytes — neither half alone would hold the claim. Test-Plan mutant #3 (replay branch named pr-N/replay) was not re-driven as a mutant; the live branch name is exercised in every H/G replay cell.
Targeted gates
Witness evidence/07-vitest-gates-head-vs-base.png; logs raw/vitest-*.log, JSON counts raw/vitest-{head,base}-4suites.json.
| Gate | Result |
|---|---|
| vitest head: resolve + autofix + ci-flaky-rerun + workflow-size | 483/483 pass (resolve 48, autofix 229, ci-flaky 8, size 198; JSON-reporter counts). First batch run hit one 30 s timeout in the autofix suite — see Environmental note |
Same 4 suites at base (HEAD^1) |
464/464 pass → delta +19, all in the resolve file (base 29 → head 48); the other three files byte-identical test counts on both arms |
qwen-pr-review-workflow.test.js (wider suite for this file) |
181/181 both arms (was 180 last round — main added one test; identical on both arms, no regression) |
bash -n on all 9 extracted blocks × both arms |
18/18 OK |
shellcheck 0.11.0 (repo-pinned) -s bash --severity=error, same 18 blocks |
18/18 OK |
| js-yaml parse of head + base workflows | 2/2 OK |
actionlint 1.7.12 (repo-pinned, via scripts/lint.js --setup) |
PR workflow clean rc=0; live-proven: planted zz-planted-job with needs: ['nonexistent-job-xyz'] → rc≠0 naming the job; restored → rc=0 |
| Size ratchet | see central-claim table (green as wired; failure direction live; fix cell live) |
| Flakiness: resolve suite × 5 identical rounds at head | PPPPP — 5 × 48/48, no divergence |
| Reviewer Test Plan walkthrough | step 1 (vitest): executed — counts differ from the claim, see Corr-1; step 2 (bash -n + YAML parse): pass; step 3 (mutation checks): all three named mutants executed and red (M-FORCE, M-FORKCHECK, branch-name shape covered by the live replay cells); step 4 (live moved-head run on the default branch): not executable here — see Not covered |
Findings
F2-drift (nit, one-character fix). .github/workflows/.size-baseline records 184442 qwen-code-pr-review.yml; the measured file at the verified head is 184,443 bytes — gap 1 byte. Round-3's F2-recurrence (1,321 bytes) is fixed: the re-record commit matched the file as it then stood. The residual byte arrived with one of the four merges of main after the re-record, and main itself carries the same unrecorded +1 byte at the base tip (file 173,140 vs entry 173,139, both frozen at HEAD^1) — so this PR inherits the drift rather than originating it, and the ratchet's allowance (4,096) keeps the gate green and silent (the gate has no branch that reports sub-allowance overruns). Practical impact today: negligible (4,095 bytes of headroom remain). Suggested fix, measured: bump the entry to 184443 — applied in a scratch tree the gate reports zero diagnostics for the file (fix cell above), while the base-baseline cell proves the gate still bites on real growth. A maintainer merging through the GitHub UI could also fix the baseline in the merge commit.
Environmental note (not a finding, counted as an assertion failure for bookkeeping). The first head batch run of the 4 suites reported 1 failed | 482 passed: qwen-autofix-workflow.test.js > behaviorally replays the stale-duplicate revalidation… timed out at its 30 s testTimeout. Attribution: the file is not touched by this PR (diff covers 3 files, none of them autofix); the batch ran while the mutation runner held a concurrent vitest in a scratch worktree; an isolated rerun of the autofix suite passed 229/229 and the clean 4-suite batch passed 483/483 (both logs in raw/). This is load-induced harness contention, not PR behaviour — recorded here so the fail count stays sacred.
Corrections
Corr-1 (re-measured, stands). The description's test counts remain stale (written before later commits added tests): it says "3 files, 267 tests (36 in resolve; 7 new)"; measured at the verified head: 3 files 285 (resolve 48, autofix 229, ci-flaky 8), resolve grew 29 → 48 = 19 new vs base. This is a correction to the description — not a request to change the code.
Injection scan: no steering instructions found in the PR title, body, or commit messages (PR text is treated as untrusted input per the verification protocol).
Not covered
- Live moved-head run on the default branch (Test Plan step 4) — requires the merged workflow. The fixture-level E2E (B1/H1–H6) is the substitute. A second stale-lease decline after replay reduces to the same re-classification path as H4 (classifier cells + code trace; the wiring attempts replay exactly once, no loop).
- Calibration against a real emitted replay comment — no GitHub token in this sandbox; the replay-success comment remains uncalibrated. Substitute (same as previous rounds): cell B1's base comment reproduces the description's quoted "Before" text byte-for-byte.
- Per-commit exercise — only 1 of the snapshot's 22 commits is object-reachable at depth 2; verified instead the aggregate
HEAD^1..HEADdiff plus the emptygit diff HEAD^2..HEAD. - fix(ci): run /resolve without the container sandbox and pin its CLI version #10428's scrub itself — still not in the new base's workflow (grep confirms no scrub step at
HEAD^1either); the scrub state was emulated exactly as the delta's commits describe (rm .git/config, no identity anywhere,GIT_CONFIG_GLOBAL=/dev/null). - yamllint —
pip3: Permission deniedin this container (fourth round running into this); YAML well-formedness covered by js-yaml parse ×2 + live-proven actionlint. shellcheck run at error severity only. - upload-artifact SHA↔tag dereference (
043fb46…as v7.0.1) — no network in the sandbox; byte-identical to the pin the two pre-existing uploads in the same file already use in production. - Round-3's observational probes not re-run this round: the degenerate replayed-tree-equals-base shape (empty
pushed.diff) and the client-side-lease probe (the latter re-confirmed anyway by H4's hook-invocation count). The loss statistics in the description (author claims; no API to check) and theShow run artifacts/Report skipped requeststeps beyond bash -n/shellcheck and their suite pins.
Methodology
Container: the verify lane's own node:22-bookworm sample (node v22.23.2, git 2.39.5, jq, bash 5.2; vitest 3.2.7). All workflow code was extracted verbatim via js-yaml from the head and base YAMLs (never retyped); ${{ }} expressions substituted exactly as Actions substitutes them (a missing wiring throws in the harness). The unit under test was never stubbed: the git seams are two bare repos reached through url.insteadOf rewrites of the production push URL injected via GIT_CONFIG_* env (applies with .git/config deleted — the state under test), real --force-with-lease declines and real pre-receive hooks produce the classifier's input bytes, and a gh stub captures exact comment bodies / scripts API sequences. Two empirical git facts shaped the harness and are worth recording: (1) a stale --force-with-lease is declined client-side before pack transfer, so a target hook never sees it (H4 measured 1 hook invocation across two pushes); (2) git show <rev>:<path> glob-matches the path portion (sha:a[1].txt silently returns the sibling a1.txt when the literal is absent), while git cat-file is literal — the assertions use cat-file. Harnesses (lib/util.mjs, lib/world.mjs, lib/stubs.mjs, ab-pushed.mjs, ab-classifier.mjs, ab-literal.mjs, ab-secondary.mjs, run-mutations.mjs, gates-size.mjs, gates-syntax.mjs, gates-vitest.mjs) and raw per-cell logs live in the artifact dir (raw/). The base arm reused the root node_modules — a clean control because the five workflow-test files import only node builtins, yaml, and a local helper (no @qwen-code/* workspace links in the closure), and the PR touches no dependencies. Assertion totals: 51 pushed-A/B + 41 classifier + 18 literal + 33 secondary + 12 mutation outcomes + 9 size cells + 41 syntax checks + 11 vitest-gate checks = 216 (214 pass, 2 fail — see Findings).
Flakiness gate log
rounds=5 files=1 skipped=0
file scripts/tests/qwen-resolve-workflow.test.js: (cd .) npx --no-install vitest run --config ./scripts/tests/vitest.config.ts ./scripts/tests/qwen-resolve-workflow.test.js
per-file results (P=pass F=fail I=infra-exit, one letter per run):
scripts/tests/qwen-resolve-workflow.test.js: PPPPP
verdict: pass
summary: 1 changed test file(s) x 5 identical rounds, no divergence
--- per-invocation detail (full copy in the artifact) ---
round 1 · scripts/tests/qwen-resolve-workflow.test.js: P (exit 0)
round 2 · scripts/tests/qwen-resolve-workflow.test.js: P (exit 0)
round 3 · scripts/tests/qwen-resolve-workflow.test.js: P (exit 0)
round 4 · scripts/tests/qwen-resolve-workflow.test.js: P (exit 0)
round 5 · scripts/tests/qwen-resolve-workflow.test.js: P (exit 0)
Evidence images
Harness scripts and raw logs are in the workflow run artifacts (7-day retention).
— Qwen Code · sandboxed verification
|
@qwen-code /resolve |
Stale bot review on a superseded commit — its blocking findings are resolved at head a0ac440 (BASE_REF added to the step env and pinned by test; size baseline re-recorded) or were recorded as non-blocking deferrals in later rounds. Dismissing my own outdated review so it does not contradict the approval pinned to a0ac440. / 针对已被取代提交的过期机器人评审——其阻断项已在 a0ac440 修复或记为非阻断延后项，予以撤销。
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship. ✅
|
Qwen Code resolved the merge conflicts and pushed the branch update. Root cause. main's #10428 ( Semantic, not textual. The step now lives in HEAD_REF: '${{ needs.resolve-pr.outputs.head_ref }}'
HEAD_SHA: '${{ needs.resolve-pr.outputs.head_sha }}'
HEAD_REPO: '${{ needs.resolve-pr.outputs.head_repo }}'
BASE_REF: '${{ needs.resolve-pr.outputs.base_ref }}'
RESOLVE_ORIGIN_URL: ''The push keeps the PR's push_resolution() {
git push --no-verify \
--force-with-lease="refs/heads/${HEAD_REF}:${HEAD_SHA}" \
"https://x-access-token:${PUSH_TOKEN}@github.com/${HEAD_REPO}.git" \
"HEAD:refs/heads/${HEAD_REF}" 2> "$push_log"
}
Load-bearing.
Could not verify. No build/tests ran. The auto-merged, NON-conflicted 中文说明根因：main 的 #10428 把 /resolve 拆为无 token 的 处理：语义冲突。env 改从 关键点：重放须留在 未验证：未跑构建/测试。未冲突的测试文件仍从 |
cde150c
…e split re-home The conflict resolution in cde150c re-homed the replay into publish-resolution in the workflow but left the 'recovering requests that used to be lost' test block sourcing `reportStep` (and the pushed-tree upload, and the env-coverage walk) from resolve-pr, where 'Report result' no longer exists — so reportStep was '' and every replay assertion and fixture in that block failed. Source the replay/push/upload assertions from publish-resolution, keep prepare/authorize on resolve-pr, walk both jobs in the uppercase-variable env-coverage test (the replay's BASE_REF lives in publish now, from needs.resolve-pr.outputs), and match the pushed-tree artifact name (needs.resolve-pr.outputs.pr_number) and its always() gate regardless of quote style. 53 pass; mutation-checked (breaking the lease move or the untouched-file guard in replay_on_moved_head reds the fixtures).
|
@qwen-code /triage |
|
Sandboxed verification: ✅ passed — merge-ready (agent verdict) - workflow run Ran the PR in an isolated, token-free container: A/B against the base build, mock-free harness assertions, targeted gates. Advisory evidence for human reviewers — not a review, an approval, or a CI check. Scripted assertions: 282 passed · 0 failed · 282 total Flakiness gate: ✅ 1 changed test file(s) x 5 identical rounds, no divergence 中文 — 判定：✅ 通过 · 可合入（agent 判定）沙箱验证在隔离、无凭证的容器中执行了该 PR 的代码（与 base 构建 A/B 对照、无 mock harness 断言、定向门禁）。仅作为评审证据，不构成评审、批准或 CI 检查。 脚本断言：282 通过 · 0 失败 · 282 总计 抖动门：✅ 1 changed test file(s) x 5 identical rounds, no divergence Verification reportPR 10429 deep verification (fifth follow-up round) —
|
| # | Finding | Severity | Status at new head |
|---|---|---|---|
| F2 / F2-drift | .size-baseline lags the actual workflow size (1,321 B, then re-drifted 1 B) |
low / nit | fixed. Re-recorded entry now matches exactly: actual 205,562 = recorded 205,562, gap 0 (witness 06-size-ratchet-cells.png). The baseline diff is one line, 193208 → 205562, matching the merged file. Note: the base tip itself carries pre-existing unrecorded growth (see Findings) |
| F3 | Under the scrub state, the replay died identity-less | medium | fixed — holds. Cell H1 runs the full block on a credential-free, identity-free publish-shaped checkout (the new job's persist-credentials: false shape): exit 0, replay pushed, committer qwen-code-dev-bot <qwen-code-dev-bot@users.noreply.github.com> |
| F1 | Clean-merge replay comment overstates | low | fixed — holds. Cell H5: conditional clause present in the archived comment; pushed.diff proves nothing came from the agent merge |
| Corr-1 | Description test counts off | nit | stands. Re-measured at the new head — see Corrections |
| M-PAREN | Parenthesised-lease guard unpinned | nit | fixed — holds. Mutant M-BAREWORD (parenthesised+anchored → bare words) killed by classifies a moved head by its lease decline… |
| L13 | Branch named x(non-fast-forward) + genuine 403 classified moved |
nit | fixed — holds. Three-arm matrix on real hook-decline logs: C6/C7 head permission, intermediate control (anchor removed) moved, base permission |
Scope selection
This round's delta (commits cde150c9 + 987d90c4): the moved-head replay re-homed from resolve-pr > Report result (old structure) into publish-resolution > Report result after #10428's job split landed on main — BASE_REF rewired through needs.resolve-pr.outputs.base_ref, fetch-by-URL via the RESOLVE_ORIGIN_URL hook, committer identity exported inside the replay, Upload pushed tree step added. Verified as the central claim below (A/B cells B1/H1–H6 against the extracted publish-resolution block).
Carried claims, all re-measured because the input closure changed (workflow re-structured, tests re-homed, new base): (1) classifier moved-before-permission ordering with parenthesised, end-of-line-anchored reasons; (2) literal pathspecs in the replay's per-file git commands; (3) authorize retry + fork preflight conjunction + draft-gate flip + deletion replay + clean-merge give-up shapes. Explicitly out of scope: the live moved-head run on the default branch, the loss statistics in the description (author claims, no API), org-owned forks (declared out of scope by the PR).
Central claim — the re-homed moved-head replay (A/B, re-measured)
The entire publish-resolution > Report result run block, js-yaml-extracted from head and base and ${{ }}-substituted exactly as Actions substitutes it, driven under bash --noprofile --norc with its own set -euo pipefail against a two-bare-repo wire (origin holding refs/pull/1/head, fork push target reached through the production-shaped x-access-token URL via url.insteadOf env rewrites), a gh stub capturing the comment body, and the publish-job checkout shape (persist-credentials: false analogue: origin remote, no credentials, no identity anywhere). The fixture seeds address-summary.md into WORKDIR exactly as the real artifact download does (Resolution check admits outcome=fixed only with it). Harness ab-pushed.mjs; witness evidence/01-ab-pushed-base-vs-head.png; per-cell logs raw/cells/.
| Cell | Arm · scenario | Result |
|---|---|---|
| B1 | base · unrelated move | exit 1 — calibration cell. Comment line byte-matches the description's quoted "Before" sentence plus the artifact pointer; fork untouched (still the moved head); no pushed/ record; no replay branch; runner left on the original resolution |
| H1 | head · unrelated move | exit 0 — the claim. Fork branch = replay commit, parents [moved head, origin/main]; committer = bot identity; author + message reused from the agent commit; tree = agent a.txt + contributor b.txt + base c.txt; pushed.sha == pushed commit; pushed.diff byte-equals an independent two-dot recompute and carries the agent resolution; comment names the moved head, points at qwen-resolve-pr-1-pushed, carries the conditional clause and the original-artifact clarification; no token in stdout/stderr/comment; fetch-by-URL hook honored (rc 0 is itself the proof — no insteadOf exists for the default github.com URL) |
| H2 | head · head did NOT move | exit 0 — direct push of the original resolution; comment byte-exact plain wording; no pushed/ record |
| H3 | head · move edits the conflicted file | exit 1 — give-up names a.txt and the original head; fork NOT rewritten; no pushed/ record; moved comment byte-exact; runner restored clean on qwen-resolve/pr-1 at the original resolution |
| H4 | head · unrelated move · fork hook declines the replay push | exit 1 — exactly 1 hook invocation (the stale first push is declined client-side before pack transfer, measured); re-classified from the replay push log (reason=permission); permission comment; fork untouched; no pushed/ record |
| H5 | head · clean-merge replay (move adopts the base side) | exit 0 — replay merged clean, took nothing from the agent's merge (pushed.diff proves it); conditional clause present (F1 holds) |
| H6 | head · identity present in .git/config |
exit 0 — replay works with config intact; committer still the exported bot identity |
Counts: 7 cells, 60 scripted oracles, 60 pass (expected base failure encoded as a passing assertion).
Secondary claim — classifier three-arm matrix on real logs
ab-classifier.mjs extracts classify_push_failure from head, the inline chain from base, and the named intermediate control (head minus the end-of-line anchor — a one-character surgical revert). Input bytes are real git push output — real stale --force-with-lease declines (remote advanced out-of-band so the lease is genuinely stale), a real (fetch first) through a second clone the first client has never fetched, real silent pre-receive-hook declines on branches named x(non-fast-forward) / x(non-fast-forward)y / fix/non-fast-forward-retry — with branch names chosen to echo classifier keywords. Only the HTTP-403 fatal line is synthetic (no HTTP status exists locally); labeled. Witness evidence/02-classifier-three-arm-real-logs.png; logs archived at raw/real-push-logs/.
| Check | head | intermediate (no anchor) | base | Note |
|---|---|---|---|---|
C1 stale · feature |
moved | moved | moved | real |
C2 stale · fix/permission-prompt |
moved | moved | permission | base loses the request |
C3 stale · fix-403-error |
moved | moved | permission | base loses the request |
C4 stale · fix/non-fast-forward-retry |
moved | moved | moved | base saved by bare-word echo |
C5 real (fetch first) |
moved | moved | moved | real |
C6 hook · x(non-fast-forward) + 403 |
permission | moved | permission | the L13 shape |
C7 hook · x(non-fast-forward)y + 403 |
permission | moved | permission | mid-name parens |
C8 hook · fix/non-fast-forward-retry + 403 |
permission | permission | permission | bare-word echo, no parens |
C9 hook phrase refusing to allow … workflow |
workflow_scope | workflow_scope | workflow_scope | priority preserved |
| C10 pure 403 / C11 unknown / C12 scope-beats-moved | permission / other / workflow_scope | same | same | |
| C13 re-classification on head | first=moved → second=permission | — | — | real first log, 403 second |
15/15 pass.
Secondary claim — literal pathspecs, head vs one-line-revert control
ab-literal.mjs drives the full extracted block with a conflicted file named a[1].txt beside its wildcard sibling a1.txt; control = head with only export GIT_LITERAL_PATHSPECS=1 removed. Witness evidence/03-literal-pathspec-ab.png.
| Cell | Scenario | head | control |
|---|---|---|---|
| G1/G1c | move touches ONLY the sibling | exit 0 — agent resolution + sibling edit both intact in the pushed tree | exit 1 — spurious give-up, wrong reason (blames the untouched conflicted file); request lost |
| G2/G2c | move edits the conflicted file | exit 1 — correct reason | exit 1 — correct outcome (union match still sees the change) |
| G3/G3c | deletion resolution + sibling move | exit 0 — file stays deleted, sibling alive | exit 1 — gives up |
15/15 pass. Blast radius bounded by construction: every control give-up fired before any commit/push — lost requests, never wrong trees.
Carried claims — authorize retry + prepare preflights (re-measured)
ab-secondary.mjs — witness evidence/04-carried-claims-secondary.png. 39/39 pass.
- Authorize retry (scripted
gh, sleep stubbed): head: 503,503,write → allowed, 3 calls, backoff 5s/10s; 503×3 → denied fail-closed after exactly 3, sleeps[5,10]; 404 → denied on the first call, no sleep; connection-reset ×2 → retried then allowed; HTTP 500 → retried (2 calls, one 5s sleep);readdenied,adminallowed. Base arm flip: first 503 denies immediately, 1 call — the loss this PR fixes. - Prepare preflights (whole extracted block, real bare origin): fork PR with edits off →
unsupportedbefore any agent work (context.md never written; reason names the setting) while base lets the same PR through withdecision=runand starts writing context.md — the wasted agent run this PR eliminates; same-repo PR withmaintainerCanModify=falseis NOT refused (conjunction holds); draft same-repo PR →decision=runat head while base still refuses it ("is draft") — the flip; fork with edits on runs; closed skips; deleted head repo → unsupported; head'sgh pr viewasks formaintainerCanModifyand no longer asks forisDraft(base: the opposite);base_ref/head_shaoutputs written for the publish job.
Mutation matrix — 13 mutants, 13 killed, 0 survivors
Scratch worktree at HEAD; unmutated control 53/53 green first; each mutant is ONE surgical YAML edit (occurrence-count asserted), oracle the resolve suite. Witness evidence/05-mutation-matrix-13-killed.png; per-mutant logs raw/mutations/.
| Mutant | Outcome | Killed by (intended assertion) |
|---|---|---|
M-PUSHED-REC remove the pushed/ recording lines |
killed | wiring pin wires the replay into the push path… |
M-PUSHED-POS move Upload pushed tree before Report result |
killed | wiring pin (position) |
M-PUSHED-GATE drop always() from its gate |
killed | wiring pin (gate string) |
M-PUSHED-NAME drop the -pushed suffix |
killed | wiring pin (name) |
M-LITERAL remove export GIT_LITERAL_PATHSPECS=1 |
killed | both glob-sibling fixture tests, behavioural — the exact spurious give-up measured in G1c |
M-ANCHOR drop the $ anchor |
killed | branch-echo classifyLog case |
| M-BAREWORD parenthesised+anchored → bare words | killed | same classifyLog case (M-PAREN gap stays pinned) |
| M-COMMENT-PUSHED-V2 delete the whole pushed-artifact sentence of the replay comment | killed | wiring pin + says in the replay comment what the run artifact does and does not describe (2 tests). Its weaker first variant (rewording the sentence opener only) survived correctly: the edit kept both substrings the pins grep — an invalid mutant, not a coverage gap; escalated and killed |
| M-SLEEP backoff 5s → 7s (positive control) | killed | the retry tests (same file) |
M-FORCE lease → --force (PR's Test-Plan mutant #1) |
killed | wires the replay… + the scope-guard pin (3 tests) |
M-FORKCHECK drop the $head_repo != $REPO half (Test-Plan mutant #2) |
killed | refuses fork PRs the bot cannot push to before spending an agent run |
M-BRANCHNAME replay branch pr-N/replay (Test-Plan mutant #3) |
killed | fixture tests fail with exactly Replay gave up: could not check out … — git refuses a ref under the existing qwen-resolve/pr-N branch (6 tests) |
M-BASE-REF drop the BASE_REF env wiring from publish Report result |
killed | defines every uppercase variable the resolve-pr and publish run blocks expand — the re-homed wiring is pinned |
The pushed-recording pins are text-level by construction; the E2E cell H1 supplies the behavioural proof that the pinned lines execute and produce the right bytes — neither half alone would hold the claim.
Targeted gates
Witness evidence/07-vitest-gates-head-vs-base.png and evidence/08-syntax-gates-and-actionlint-live.png; logs raw/vitest-*.log, JSON counts raw/vitest-gates.json.
| Gate | Result |
|---|---|
| vitest head: resolve + autofix + ci-flaky-rerun + workflow-size | 488/488 pass (resolve 53, autofix 229, ci-flaky 8, size 198; JSON-reporter counts) |
Same 4 suites at base (HEAD^1) |
469/469 pass → delta +19, all in the resolve file (base 34 → head 53); the other three files byte-identical test counts on both arms |
qwen-pr-review-workflow.test.js (wider suite for this file) |
181/181 both arms — no regression on the wider surface |
bash -n on all 30 extracted run blocks × both arms |
60/60 OK |
shellcheck 0.11.0 (repo-pinned) -s bash --severity=error, same 60 blocks |
60/60 OK |
| js-yaml parse of head + base workflows | 2/2 OK |
actionlint (repo-pinned, via scripts/lint.js --setup) |
all workflows clean rc=0; live-proven: planted zz-planted-job with needs: [nonexistent-job-xyz] → rc=1 naming the job at line 3556; restored → rc=0 |
| Size ratchet | actual = recorded = 205,562, gap 0; gate as CI wires it (no WORKFLOW_SIZE_BASE_SHA) green and silent; failure direction live (scratch tree with the base entry 193,208 → exit 1, grew to 205562 bytes, 12354 over its recorded 193208); fix cell green |
| Flakiness: resolve suite × 5 identical rounds at head | PPPPP — 5 × 53/53, no divergence |
| Reviewer Test Plan walkthrough | step 1 (vitest): executed — counts differ from the claim, see Corr-1; step 2 (bash -n + YAML parse): pass; step 3 (mutation checks): all three named mutants executed and red (M-FORCE, M-FORKCHECK, M-BRANCHNAME with the exact predicted could not check out); step 4 (live moved-head run on the default branch): not executable here — see Not covered |
Findings
No blocking defect and no new defect in the PR's code were found this round; every executed assertion passed. Two informational items, both outside the PR's changeset:
- (nit, pre-existing on main, NOT introduced by this PR) The base tip
HEAD^1itself carries unrecorded workflow growth: the file measures 194,395 bytes there against its recorded 193,208 — gap 1,187 bytes, inside the ratchet's 4,096 allowance, so the gate is green and silent (it has no branch that reports sub-allowance overruns). This is main-side drift the fix(ci): run /resolve without the container sandbox and pin its CLI version #10428 split left behind; this PR's own re-record is exact (gap 0 at the verified head), so the PR inherits nothing and leaves nothing. A maintainer may want to re-record the baseline on main at some point; nothing in this PR needs to change. - (note) The fix(ci): run /resolve without the container sandbox and pin its CLI version #10428 post-agent scrub step is still not present in this base's workflow (grep confirms no scrub step at
HEAD^1); the job split supersedes most of its purpose for the publish path —publish-resolutionstarts from a freshactions/checkoutwithpersist-credentials: false, so the replay's credential-free assumptions hold by construction. The replay's defensive measures (fetch-by-URL, exported committer identity) were exercised in exactly that state (cell H1) and remain valid if a scrub lands later.
Corrections
Corr-1 (re-measured, stands). The description's test counts remain stale: it says "3 files, 267 tests (36 in resolve; 7 new)"; measured at the verified head: 3 files 290 (resolve 53, autofix 229, ci-flaky 8), resolve grew 34 → 53 = 19 new vs base. This is a correction to the description — not a request to change the code.
Injection scan: no steering instructions found in the PR title, body, or commit messages (PR text is treated as untrusted input per the verification protocol).
Not covered
- Live moved-head run on the default branch (Test Plan step 4) — requires the merged workflow. The fixture-level E2E (B1/H1–H6) is the substitute. A second stale-lease decline after replay reduces to the same re-classification path as H4 (classifier cells C13 + code trace; the wiring attempts replay exactly once, no loop). This round additionally reproduces the wire shape, not the trigger: the decline/replay bytes are real git output produced locally, not a recording of a production decline.
- Calibration against a real emitted replay comment — no GitHub token in this sandbox; the replay-success comment remains uncalibrated against production. Substitute (same as previous rounds): cell B1's base comment reproduces the description's quoted "Before" text byte-for-byte.
- Per-commit exercise — only 1 of the snapshot's 23 commits is object-reachable at depth 2; verified instead the aggregate
HEAD^1..HEADdiff, and confirmed the non-emptygit diff HEAD^2..HEADis exactly the base-side test: harden release timing-sensitive waits #10615 change (two test files untouched by this PR), so the merge introduced nothing PR-adjacent. - yamllint —
pip3: Permission deniedin this container (fifth round running into this); YAML well-formedness covered by js-yaml parse ×2 + live-proven actionlint. shellcheck run at error severity only. - upload-artifact SHA↔tag dereference (
043fb46…as v7.0.1) — no network in the sandbox; byte-identical to the pin the two pre-existing uploads in the same file already use at base in production. - The loss statistics in the description (294 resolutions, 17 lost to moved heads, 10 fork runs, 182 draft skips — author claims; no API to check) and the
Show run artifacts/Report skipped requeststeps beyond bash -n/shellcheck and their suite pins.
Methodology
Container: the verify lane's own node:22-bookworm sample (node v22.23.2, git 2.39.5, bash 5.2, vitest 3.2.7). All workflow code was extracted verbatim via js-yaml from the head and base YAMLs (never retyped); ${{ }} expressions substituted exactly as Actions substitutes them (a missing wiring throws in the harness). The unit under test was never stubbed: the git seams are two bare repos reached through url.insteadOf rewrites of the production push URL injected via GIT_CONFIG_* env (applies with no credentials in .git/config — the state under test), real --force-with-lease declines and real pre-receive hooks produce the classifier's input bytes, and a gh stub captures exact comment bodies. Three empirical git facts shaped the harness and are worth recording: (1) a stale --force-with-lease is declined client-side before pack transfer, so a target hook never sees it (H4 measured 1 hook invocation across two pushes); (2) git cat-file treats rev:path literally while git show glob-matches the path portion — assertions use cat-file; (3) in this container's identity-free environment git merge validates the committer before computing conflicts, so the fixture-side resolution merge carries an identity while the state under test stays identity-free. Harnesses (lib/util.mjs, lib/world.mjs, lib/stubs.mjs, ab-pushed.mjs, ab-classifier.mjs, ab-literal.mjs, ab-secondary.mjs, run-mutations.mjs, run-mutation-v2.mjs, run-mutation-branchname.mjs, run-mutation-baseref.mjs, gates-size.mjs, gates-syntax.mjs, gates-vitest.mjs) and raw per-cell logs live in the artifact dir (raw/). The base arm reused the root node_modules — a clean control because the workflow-test files import only node builtins, yaml, and a local helper (no @qwen-code/* workspace links in the closure), and the PR touches no dependencies. Assertion totals: 60 pushed-A/B + 15 classifier + 15 literal + 39 secondary + 14 mutation-matrix (control + 13 mutants) + 10 vitest + 122 syntax + 3 actionlint + 4 size-ratchet = 282 pass, 0 fail — every number maps to a scripted check that ran.
Flakiness gate log
rounds=5 files=1 skipped=0
file scripts/tests/qwen-resolve-workflow.test.js: (cd .) npx --no-install vitest run --config ./scripts/tests/vitest.config.ts ./scripts/tests/qwen-resolve-workflow.test.js
per-file results (P=pass F=fail I=infra-exit, one letter per run):
scripts/tests/qwen-resolve-workflow.test.js: PPPPP
verdict: pass
summary: 1 changed test file(s) x 5 identical rounds, no divergence
--- per-invocation detail (full copy in the artifact) ---
round 1 · scripts/tests/qwen-resolve-workflow.test.js: P (exit 0)
round 2 · scripts/tests/qwen-resolve-workflow.test.js: P (exit 0)
round 3 · scripts/tests/qwen-resolve-workflow.test.js: P (exit 0)
round 4 · scripts/tests/qwen-resolve-workflow.test.js: P (exit 0)
round 5 · scripts/tests/qwen-resolve-workflow.test.js: P (exit 0)
Evidence images
Harness scripts and raw logs are in the workflow run artifacts (7-day retention).
— Qwen Code · sandboxed verification
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship. ✅
Same conflict as 76acdcd, same resolution: both sides re-recorded qwen-code-pr-review.yml in .size-baseline (branch 198946 after the previous merge, main 205562 after #10429's replay machinery); the entry is the merged file's actual size (210113), verified with check-workflow-size.sh on the merged tree.
Main split the /resolve lane into resolve-pr and publish-resolution (#10428, #10429). Resolve the HELPER_TESTS conflict as the union of both new entries, extend the health watch's producer pins to both jobs, and classify the lane's new lost-artifact report sentence as infra_failed instead of letting it fall through to unknown. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
|
Released in v0.23.0. |
4 participants
What this PR does
Four changes to the
/resolvepath inqwen-code-pr-review.yml, each recovering a class of request that the survey behind #10428 showed being lost after the agent had already done its work, or before it was allowed to start:Report resultnow fetches the new head, redoes the merge of the base on top of it, and for every file that still conflicts takes the agent's resolution if the new commits did not touch that file. The replayed tree is re-checked with the same structural guardsResolution checkapplied to the first merge (no conflict markers, merges cleanly against the base, only base-changed files), committed with the agent's message, and pushed with a lease on the new head. If any still-conflicting file was changed by the new commits, the replay gives up, restores the original resolution, and the run reports "moved" exactly as before. The agent is never re-run.Prepare pull request branchreadsmaintainerCanModify; a fork PR with "Allow edits by maintainers" off gets the skip comment immediately instead of a full agent run whose push is then rejected.authorizeasks up to three times (5s, 10s back-off) whengh api …/permissionfails with a 5xx-class error before treating the error as a denial. The fail-closed outcome for a real error or a non-writer is unchanged./resolveis an explicit request from someone with write permission; the draft gate only turned such requests into a skip comment. The automatic review lane keeps its own draft gate.Tests: the replay functions are extracted from the
run:block and exercised against a real git fixture (bare origin withrefs/pull/1/head, a runner checkout holding the agent's resolution, a contributor clone that moves the head) in three cases — unrelated move → replayed on the new head with the agent's resolution, the contributor's change and the base merge all present, only base-changed files differing from the new head; move that edits the conflicted file → gives up with the tree restored to the original resolution; no move → gives up. Contract pins cover the push wiring (lease on${HEAD_SHA}, moved to the new head; no bare--force), the fork preflight (conjoined with the head-repo comparison, since same-repo PRs also reportmaintainerCanModify=false), the draft-gate removal, and the retry loop.Why it's needed
Of the 294 resolutions the agent has produced since the command shipped, 17 were thrown away because the head moved during the run — the largest single loss after the agent itself — and in the runs inspected the intervening pushes were unrelated to the conflict (a CI fix, a review nit). Ten more agent runs were spent on fork PRs whose push was then rejected for lack of maintainer edits, something the PR metadata says before the run starts. A 503 from the permission API silently denied requests from a writer with no comment on the PR (e.g. run 32054217289), and 182 explicit requests on draft PRs were answered with "PR is draft". Together these are most of the non-agent losses once the outage fixed by #10428 is out of the way.
Reviewer Test Plan
How to verify
npx vitest run --config ./scripts/tests/vitest.config.ts scripts/tests/qwen-resolve-workflow.test.js scripts/tests/qwen-autofix-workflow.test.js scripts/tests/ci-flaky-rerun-workflow.test.js→ 3 files, 267 tests pass (36 in the resolve file; 7 new, 3 of them driving real git repositories).run:blocks (authorizepermission check,Prepare pull request branch,Report result) passbash -nafter stripping${{ }}expressions; the workflow parses with js-yaml.--forceinstead of the lease → "wires the replay…" fails; dropping the[ "$head_repo" != "$REPO" ]half of the fork check → "refuses fork PRs…" fails; naming the replay branchqwen-resolve/pr-N/replay(a ref under the existingqwen-resolve/pr-Nbranch, which git refuses) → the fixture test fails with "could not check out" — this is how the currentpr-N-replayname was arrived at.@qwen-code /resolveand push an unrelated commit to the head while the agent runs; expect the result comment to say the head moved to<sha>and the resolution was replayed, and the pushed merge commit to have that<sha>as its first parent.Evidence (Before & After)
Before — the "moved" outcome as it has read on 17 PRs:
Qwen Code resolved the merge conflicts, but the head branch changed while resolving, so the update was not pushed. Re-run /resolve.— the resolution exists only as an artifact and the re-run costs a second agent run.After — fixture run from the new test, unrelated move (b.txt) while a.txt was the conflicted file:
Move that edits a.txt:
rc=1,Replay gave up: a.txt still conflicts and the new head changed it since <original head>, checkout back onqwen-resolve/pr-1at the original resolution, clean tree.Tested on
Environment (optional)
Unit tests only (vitest with real git fixtures).
Risk & Scope
/resolverun instead of a skip comment.Linked Issues
Related: #10428 (restores the lane this PR builds on).
中文说明
本 PR 做了什么
对
qwen-code-pr-review.yml中/resolve链路做四处改动，每一处都对应 #10428 背后的统计里发现的一类损失——agent 已经完成工作却被丢弃，或者根本没被允许开始：Report result现在会拉取新 head，在其之上重新合并 base，对每个仍然冲突的文件，如果新提交没有触碰该文件，就采用 agent 的解决版本。重放后的树会用Resolution check对第一次合并所做的同样结构检查再验一遍（无冲突标记、能干净合并到 base、只改动 base 变更过的文件），用 agent 的提交信息提交，然后以新 head 为 lease 推送。若任何仍冲突的文件被新提交改过，重放放弃，恢复原来的解决结果，run 与以前一样报告 "moved"。agent 不会被重新运行。Prepare pull request branch读取maintainerCanModify；来自 fork 且未勾选 "Allow edits by maintainers" 的 PR 直接得到跳过评论，而不是跑完一整轮 agent 再在推送时被拒。authorize在gh api …/permission返回 5xx 类错误时最多重试三次（退避 5s、10s），然后才把错误当作拒绝。真实错误或非写权限用户的 fail-closed 结果不变。/resolve是有写权限的人发出的显式请求；draft 门只是把这类请求变成一条跳过评论。自动评审链路保留自己的 draft 门。测试：从
run:块中提取重放函数，在真实 git fixture 上运行（带refs/pull/1/head的裸 origin、持有 agent 解决结果的 runner 检出、能移动 head 的贡献者克隆），覆盖三种情况——无关移动 → 在新 head 上重放，agent 的解决、贡献者的新改动、base 合并全部在场，与新 head 的差异只有 base 变更过的文件；移动改到了冲突文件 → 放弃并把树恢复到原来的解决结果；没有移动 → 放弃。契约钉子覆盖推送接线（lease 仍用${HEAD_SHA}、重放后指向新 head；没有裸--force）、fork 预检（必须与 head 仓库比较联立，因为同仓 PR 的maintainerCanModify也是 false）、draft 门的移除、以及重试循环。为什么需要
自命令上线以来 agent 产出的 294 次解决结果中，有 17 次因为 head 在运行期间变动而被丢弃——这是 agent 自身之外最大的单项损失——而抽查的这些 run 里，中间的推送都与冲突无关（CI 修复、评审小改）。另有 10 次 agent 运行花在 fork PR 上，推送时因没有 maintainer edit 权限被拒，而这一点在运行前从 PR 元数据就能知道。权限 API 的一次 503 会静默拒绝有写权限者的请求且 PR 上没有任何评论（如 run 32054217289），draft PR 上的 182 次显式请求得到的回复是 "PR is draft"。在 #10428 修复的故障之外，这些占了非 agent 原因损失的大部分。
评审验证方案
如何验证
npx vitest run --config ./scripts/tests/vitest.config.ts scripts/tests/qwen-resolve-workflow.test.js scripts/tests/qwen-autofix-workflow.test.js scripts/tests/ci-flaky-rerun-workflow.test.js→ 3 个文件 267 个测试通过（resolve 文件 36 个；新增 7 个，其中 3 个驱动真实 git 仓库）。run:块（authorize权限检查、Prepare pull request branch、Report result）去掉${{ }}表达式后通过bash -n；workflow 能被 js-yaml 解析。--force而非 lease 推送 → "wires the replay…" 失败；去掉 fork 检查里[ "$head_repo" != "$REPO" ]这一半 → "refuses fork PRs…" 失败；把重放分支命名为qwen-resolve/pr-N/replay（位于已存在的qwen-resolve/pr-N分支之下的 ref，git 会拒绝）→ fixture 测试报 "could not check out"——当前的pr-N-replay命名就是这样得出的。@qwen-code /resolve，并在 agent 运行期间向 head 推一个无关提交；预期结果评论说明 head 移动到了<sha>且解决结果已重放，推送的合并提交的第一个父提交是该<sha>。证据（改动前后）
改动前——17 个 PR 上出现过的 "moved" 结果：
Qwen Code resolved the merge conflicts, but the head branch changed while resolving, so the update was not pushed. Re-run /resolve.——解决结果只存在于 artifact 里，重跑要再花一轮 agent。改动后——新测试的 fixture 运行，冲突文件是 a.txt，期间无关移动（b.txt）：
移动改到 a.txt：
rc=1，Replay gave up: a.txt still conflicts and the new head changed it since <original head>，检出回到qwen-resolve/pr-1上原来的解决结果，工作树干净。测试平台
环境（可选）
仅单元测试（vitest + 真实 git fixture）。
风险与范围
/resolve，而不是收到跳过评论。关联 Issue
相关：#10428（恢复本 PR 所依赖的链路）。