-
Notifications
You must be signed in to change notification settings - Fork 3k
feat(autofix): escalate a non-converging diff to a maintainer handoff #9104
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -165,6 +165,15 @@ env: | |
| # value falls back to its default at the read site. | ||
| GROWTH_BUDGET_SRC_LINES: '${{ vars.QWEN_AUTOFIX_GROWTH_BUDGET_SRC_LINES || 400 }}' | ||
| GROWTH_BUDGET_TEST_LINES: '${{ vars.QWEN_AUTOFIX_GROWTH_BUDGET_TEST_LINES || 400 }}' | ||
| # Non-convergence handoff: once the growth brake has been over budget for | ||
| # this many PRIOR rounds in the window AND the diff has not shrunk from the | ||
| # most recent over-budget round, the round is DIVERGING (the fixes keep | ||
| # growing the diff, so | ||
| # Critical-only — which only trims non-Criticals — cannot help). At that | ||
| # point the round escalates to a maintainer-decision handoff (split / accept | ||
| # core + track the tail / redesign) instead of patching again. Same tunable | ||
| # contract as the budgets above (malformed → default at the read site). | ||
| GROWTH_DIVERGENCE_ROUNDS: '${{ vars.QWEN_AUTOFIX_GROWTH_DIVERGENCE_ROUNDS || 2 }}' | ||
| # An auth/access model error (401/402/403, "no access"/"does not exist") | ||
| # never self-heals - only a maintainer can fix the key - and every retry | ||
| # costs an agent run AND a PR comment. Cap those attempts far below | ||
|
|
@@ -4463,6 +4472,11 @@ jobs: | |
| # round's pushed growth would escape the budget for the rest of | ||
| # the live window. | ||
| echo "growth_base_win=${LIVE_REARM_KEY}" | ||
| # The round's own growth + over-budget flag, so the report step can | ||
| # write this round's autofix-growth-now marker (the per-round | ||
| # history the divergence read above consumes). | ||
| echo "growth_src=${GROWTH_SRC}" | ||
| echo "growth_test=${GROWTH_TEST}" | ||
| } >> "${GITHUB_OUTPUT}" | ||
| echo "📏 net diff src ${NET_SRC} / test ${NET_TEST} lines (window baseline ${BASE_SRC}/${BASE_TEST}, growth ${GROWTH_SRC}/${GROWTH_TEST}, budgets ${GROWTH_BUDGET_SRC_LINES}/${GROWTH_BUDGET_TEST_LINES})" | ||
|
|
||
|
|
@@ -4477,6 +4491,74 @@ jobs: | |
| CRITICAL_ONLY='true' | ||
| CRITICAL_ONLY_GROWTH='true' | ||
| fi | ||
| # Divergence: Critical-only only trims non-Criticals, so when the | ||
| # GROWTH that trips the brake is Critical-driven the diff keeps | ||
| # climbing anyway. Read this window's prior per-round growth markers | ||
| # (written by the report step): count the rounds that were over | ||
| # budget, and take the MOST RECENT prior over-budget run's growth | ||
| # SUM (highest run_id — see below; NOT the window-wide max, which a | ||
| # one-off spike would raise forever). The round is DIVERGING when it | ||
| # is over budget now, the brake has already fired for | ||
| # >= GROWTH_DIVERGENCE_ROUNDS prior rounds, and the diff has NOT | ||
| # shrunk from that most-recent sum — the fixes are not converging, so | ||
| # the round must escalate to a human decision instead of patching | ||
| # again. A diff that is over budget but SHRINKING (agent removing | ||
| # code) or a one-off overshoot stays in ordinary Critical-only. | ||
| if [[ ! "${GROWTH_DIVERGENCE_ROUNDS}" =~ ^([1-9][0-9]{0,3})$ ]]; then | ||
| echo "::warning::GROWTH_DIVERGENCE_ROUNDS='${GROWTH_DIVERGENCE_ROUNDS}' is not a positive count; using 2" | ||
| GROWTH_DIVERGENCE_ROUNDS=2 | ||
| fi | ||
|
wenshao marked this conversation as resolved.
|
||
| # Count runs whenever the net is measured (not only over budget), so | ||
| # the trajectory clause below is accurate even on a round that pulled | ||
| # back under budget. markers: | ||
| # <!-- autofix-growth-now src=N test=N over=BOOL round=N run=ID key=W --> | ||
| # Deduped and ordered by run=GITHUB_RUN_ID, the per-workflow-run id: | ||
| # the report post's bounded retry re-posts one run's marker (same | ||
| # run_id → collapses to the LATEST by created_at, so a re-run's | ||
| # fresher attempt wins over its own stale first post), while every | ||
| # distinct address run has a fresh, monotonically increasing run_id. round=/eval-watermark are NOT a | ||
| # safe identity — a state-triggered lane (a persistent merge conflict | ||
| # selects the PR every scan with no new evaluable feedback) freezes | ||
| # both NEWEST and ROUND, so distinct over-budget runs would share them | ||
| # and collapse, stalling the count. Filtered to markers AFTER any | ||
| # mid-window base update (BASE_UPD_AT) — the same re-anchoring the | ||
| # growth-base read guards against. The "not shrinking" test compares | ||
| # against the MOST RECENT prior over-budget run's sum (highest | ||
| # run_id), not the window-wide max: a single transient spike would | ||
| # otherwise raise the bar forever and a genuine plateau-over-budget | ||
| # runaway (the exact case to escalate) would never clear it. The | ||
| # CURRENT run's own markers are excluded (run != GITHUB_RUN_ID): a | ||
| # re-run of a failed job keeps the same run id and its failed | ||
| # attempt already posted a marker, so counting it would over-report | ||
| # the round's own attempt as a PRIOR one. | ||
| GROWTH_DIVERGED='false' | ||
| OVER_ROUNDS_PRIOR=0 | ||
| PREV_SUM=0 | ||
| if [[ "${NET_MEASURED}" == 'true' ]]; then | ||
| read -r OVER_ROUNDS_PRIOR PREV_SUM < <(jq -r --arg ab "${AUTOFIX_BOT}" --arg key "${LIVE_REARM_KEY}" --arg baseupd "${BASE_UPD_AT}" --arg curr "${GITHUB_RUN_ID}" ' | ||
| [ .[] | select((.user.login // "") == $ab) | . as $c | ($c.body // "") | ||
| | [ scan("<!-- autofix-growth-now src=(-?[0-9]+) test=(-?[0-9]+) over=([a-z]+) round=([0-9]+) run=([0-9]+) key=([^ ]+) -->") ] | .[] | ||
| | {sum: ((.[0] | tonumber) + (.[1] | tonumber)), over: .[2], round: (.[3] | tonumber), run: (.[4] | tonumber), win: .[5], at: ($c.created_at // "")} ] | ||
|
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Critical] R6-2: Source and test growth have separate budgets, but the detector compares only their signed sum. Failure scenario: prior source/test growth is 500/500 and current growth is 900/0; source worsens over its own budget, but the sum falls 1000→900 and suppresses the handoff. Preserve both dimensions and compare the currently over-budget dimension(s) independently. Witness: 中文说明source/test 有独立预算,但检测器只比较有符号总和。上一轮 500/500、本轮 900/0 时,source 已恶化并超预算,总和却从 1000 降至 900,错误抑制 handoff。请保留两个维度并分别判断当前超预算维度。— gpt-5.6-sol via Qwen Code /review (v0.21.11) |
||
| | map(select(.win == $key and .over == "true")) | ||
|
wenshao marked this conversation as resolved.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Suggestion] R2-8: the comparability guard covers mid-window BASE UPDATES but not AUTHOR PUSHES — and an author push shifts the net-diff measurement while nothing re-anchors. Verified at HEAD: 中文说明R2-8:可比性守卫覆盖窗口内的 base 更新,但不覆盖作者 push——而作者 push 会改变净 diff 测量,同时没有任何东西重新锚定。在 HEAD 上核实: — qwen3.8-max via Qwen Code /review (v0.21.11)
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Acknowledged, deferring with rationale. Correct that an author push mid-window can make prior sums (old head) incomparable to this round's (new head) and produce a false handoff — but the outcome is fail-safe: the handoff stops and asks a maintainer, who is looking at a genuinely large diff anyway, and
wenshao marked this conversation as resolved.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Critical] R6-4: Filtering to Witness: 中文说明排序前过滤为 `over=true`,会导致恢复轮永远无法重置发散计数。多次“超预算→恢复”后再次超预算,旧记录仍会达到阈值。请只统计最近一次恢复后的尾部连续超预算区间。— gpt-5.6-sol via Qwen Code /review (v0.21.11) |
||
| | map(select(.run != ($curr | tonumber))) | ||
| | map(select($baseupd == "" or (.at > $baseupd))) | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Suggestion] R2-6: this comparability filter uses the marker comment's 中文说明R2-6:这个可比性过滤器把 marker 评论的 — qwen3.8-max via Qwen Code /review (v0.21.11)
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Acknowledged, deferring with rationale. This is real but self-bounded: it needs a round in flight during a concurrent base update, and you note it's 'at most one round early and fails safe' (a false handoff waits for a maintainer;
wenshao marked this conversation as resolved.
|
||
| | group_by(.run) | map(max_by(.at)) | ||
| | sort_by(.run) | ||
| | "\(length) \((last.sum) // 0)"' "${WORKDIR}/ic.json" 2> /dev/null || echo "0 0") | ||
| [[ "${OVER_ROUNDS_PRIOR}" =~ ^[0-9]+$ ]] || OVER_ROUNDS_PRIOR=0 | ||
| [[ "${PREV_SUM}" =~ ^-?[0-9]+$ ]] || PREV_SUM=0 | ||
| if [[ "${CRITICAL_ONLY_GROWTH}" == 'true' \ | ||
| && "${OVER_ROUNDS_PRIOR}" -ge "${GROWTH_DIVERGENCE_ROUNDS}" \ | ||
| && $(( GROWTH_SRC + GROWTH_TEST )) -ge "${PREV_SUM}" ]]; then | ||
| GROWTH_DIVERGED='true' | ||
| fi | ||
| fi | ||
| # The over-budget flag feeds the report's per-round marker; the | ||
| # handoff itself is enforced by the feedback.md text below, so | ||
| # GROWTH_DIVERGED needs no step output. | ||
| echo "critical_only_growth=${CRITICAL_ONLY_GROWTH}" >> "${GITHUB_OUTPUT}" | ||
|
wenshao marked this conversation as resolved.
|
||
| [[ "${GROWTH_DIVERGED}" == 'true' ]] && | ||
| echo "🛑 diff not converging: over budget now, ${OVER_ROUNDS_PRIOR} prior over-budget round(s) in this window, growth not shrinking — escalating to a maintainer decision instead of patching." | ||
| # Which trusted humans have exhausted their per-window regular | ||
| # feedback budget (see CRITICAL_ONLY_HUMAN_BATCHES). A batch is | ||
| # COUNTED only when a Critical-only round actually consumed it: | ||
|
|
@@ -4655,6 +4737,26 @@ jobs: | |
| echo "Only feedback newer than the last evaluation (${WATERMARK}) from" | ||
| echo "trusted maintainers or the automated reviewer is listed." | ||
| echo | ||
| # Diff-growth trajectory: the agent sees how big this PR has grown | ||
| # so it can prefer minimal/subtractive fixes and recognise a | ||
| # non-converging spiral, rather than reflexively adding code for | ||
| # every finding. | ||
| if [[ "${NET_MEASURED}" == 'true' ]]; then | ||
| echo "## Diff growth this window" | ||
|
wenshao marked this conversation as resolved.
|
||
| echo | ||
| echo "Net diff vs this counting window's baseline: source ${GROWTH_SRC} / test ${GROWTH_TEST} lines (budgets ${GROWTH_BUDGET_SRC_LINES}/${GROWTH_BUDGET_TEST_LINES}; ${OVER_ROUNDS_PRIOR} prior round(s) already over budget). Prefer minimal, root-cause, subtractive fixes. If closing a finding would grow the diff materially AND the same class of gap keeps reappearing on code earlier rounds added, that is a signal to escalate for a split — not to keep adding guards." | ||
|
wenshao marked this conversation as resolved.
|
||
| echo | ||
| fi | ||
| # Non-convergence handoff (the diff keeps growing past budget across | ||
| # rounds): a maintainer-decision item the agent must NOT settle by | ||
| # patching. Framed as defer-to-human so the address run stops with a | ||
| # handoff (SKILL: "Stop BLOCKED when any defer-to-human item remains"). | ||
| if [[ "${GROWTH_DIVERGED}" == 'true' ]]; then | ||
|
wenshao marked this conversation as resolved.
|
||
| echo "## Needs a maintainer's decision — this PR is not converging" | ||
|
wenshao marked this conversation as resolved.
wenshao marked this conversation as resolved.
|
||
| echo | ||
| echo "This PR's diff has stayed over the growth budget for ${OVER_ROUNDS_PRIOR}+ rounds in this window and is still not shrinking (source ${GROWTH_SRC} / test ${GROWTH_TEST} net lines vs budgets ${GROWTH_BUDGET_SRC_LINES}/${GROWTH_BUDGET_TEST_LINES}). The review findings are themselves driving the growth, so continuing to patch will not converge — Critical-only mode cannot help because the Criticals are the growth. Do NOT apply more code fixes this round. Treat this as a \`defer-to-human\` item: STOP with a handoff that names the decision and lays out the options — split the PR (land the core, track the remaining findings as follow-up issues), redesign the approach, or accept the current state with the tail deferred — plus your recommendation. Leave the call to the maintainer; declining or implementing one direction yourself IS deciding." | ||
| echo | ||
| fi | ||
| echo "## Reviews" | ||
| jq -r --arg wm "${WATERMARK}" --arg rb "${REVIEW_BOT}" --arg ab "${AUTOFIX_BOT}" \ | ||
| --argjson critical_only "${CRITICAL_ONLY}" --argjson trust "${TRUSTED_ASSOC}" \ | ||
|
|
@@ -5251,6 +5353,12 @@ jobs: | |
| # can report under a stale WINDOW after a re-arm; the marker must | ||
| # land under the key later reads will use. | ||
| GROWTH_BASE_WIN: '${{ steps.prepare.outputs.growth_base_win }}' | ||
| # This round's own growth + over-budget flag, written as the | ||
| # per-round autofix-growth-now marker so later rounds can measure | ||
| # divergence (growth still climbing over budget = not converging). | ||
| GROWTH_SRC: '${{ steps.prepare.outputs.growth_src }}' | ||
| GROWTH_TEST: '${{ steps.prepare.outputs.growth_test }}' | ||
| CRITICAL_ONLY_GROWTH: '${{ steps.prepare.outputs.critical_only_growth }}' | ||
|
wenshao marked this conversation as resolved.
wenshao marked this conversation as resolved.
wenshao marked this conversation as resolved.
|
||
| run: |- | ||
| # gh has its own $GITHUB_ENV-injectable channels: pin the host and | ||
| # drop any planted token BEFORE the identity check below, so a | ||
|
|
@@ -5620,6 +5728,10 @@ jobs: | |
| if [[ "${GROWTH_BASE_NEW}" == 'true' ]]; then | ||
| echo "<!-- autofix-growth-base src=${GROWTH_BASE_SRC} test=${GROWTH_BASE_TEST} key=${GROWTH_BASE_WIN:-${WINDOW:-none}} -->" | ||
| fi | ||
| # Per-round growth history the next round's divergence read | ||
| # counts; run=GITHUB_RUN_ID is the dedup+order identity (a retry | ||
| # re-posts the same run, distinct runs get fresh run ids). | ||
| echo "<!-- autofix-growth-now src=${GROWTH_SRC:-0} test=${GROWTH_TEST:-0} over=${CRITICAL_ONLY_GROWTH:-false} round=${NEXT_ROUND} run=${GITHUB_RUN_ID} key=${GROWTH_BASE_WIN:-${WINDOW:-none}} -->" | ||
|
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Critical] R6-6: Historical markers store only Witness: 中文说明历史 marker 只保存 `over=true`,没有保存当时的有效预算。旧预算 400 下的两次 500 行记录,在预算提高到 1000 后仍会被当作新预算下的先前超标轮,导致首次 1100 行就过早 handoff。请记录并匹配有效预算,或把预算配置纳入窗口身份。— gpt-5.6-sol via Qwen Code /review (v0.21.11) |
||
| } > "${WORKDIR}/report.md" | ||
| STATUS="pushed (round ${NEXT_ROUND}/${MAX_ROUNDS})" | ||
| else | ||
|
|
@@ -5644,6 +5756,10 @@ jobs: | |
| if [[ "${GROWTH_BASE_NEW}" == 'true' ]]; then | ||
| echo "<!-- autofix-growth-base src=${GROWTH_BASE_SRC} test=${GROWTH_BASE_TEST} key=${GROWTH_BASE_WIN:-${WINDOW:-none}} -->" | ||
| fi | ||
| # Per-round growth history the next round's divergence read | ||
| # counts; run=GITHUB_RUN_ID is the dedup+order identity (a retry | ||
| # re-posts the same run, distinct runs get fresh run ids). | ||
| echo "<!-- autofix-growth-now src=${GROWTH_SRC:-0} test=${GROWTH_TEST:-0} over=${CRITICAL_ONLY_GROWTH:-false} round=${ROUND} run=${GITHUB_RUN_ID} key=${GROWTH_BASE_WIN:-${WINDOW:-none}} -->" | ||
| } > "${WORKDIR}/report.md" | ||
| STATUS="no action needed" | ||
| fi | ||
|
|
@@ -5790,6 +5906,16 @@ jobs: | |
| EFFECTIVE_ROUND: '${{ steps.prepare.outputs.effective_round }}' | ||
| MODEL: '${{ vars.QWEN_AUTOFIX_MODEL || vars.QWEN_PR_REVIEW_MODEL }}' | ||
| CHECKED_OUT_HEAD: '${{ steps.prepare.outputs.checked_out_head }}' | ||
| # This step also posts a round report (timeout / gate-rejection / | ||
| # abort), so it writes the per-round growth-now marker too — else an | ||
| # over-budget round that never reaches 'Push and report' leaves a | ||
| # history gap and the divergence count under-reports. Empty outputs | ||
| # (prepare never ran) fall through the :-0/:-false marker fallbacks | ||
| # to an inert over=false entry. | ||
| GROWTH_SRC: '${{ steps.prepare.outputs.growth_src }}' | ||
| GROWTH_TEST: '${{ steps.prepare.outputs.growth_test }}' | ||
| CRITICAL_ONLY_GROWTH: '${{ steps.prepare.outputs.critical_only_growth }}' | ||
| GROWTH_BASE_WIN: '${{ steps.prepare.outputs.growth_base_win }}' | ||
| run: |- | ||
| # The head the agent actually evaluated — captured in prepare before | ||
| # any mutation, not the report-time remote head (which can move | ||
|
|
@@ -6295,6 +6421,11 @@ jobs: | |
| echo "🧠 Handled by **Qwen Code** · model/模型 \`${MODEL_DISPLAY}\`" | ||
| echo | ||
| echo "<!-- autofix-eval ts=${MARK_TS} acted=false round=${MARK_ROUND} win=${WINDOW:-none} -->" | ||
| # Per-round growth history the divergence read counts — same | ||
| # marker the push/no-op report paths write, so an over-budget | ||
| # round that timed out or was gate-rejected is not a gap. run= | ||
|
Comment on lines
+6424
to
+6426
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Critical] R6-3: The stale-base retry can update the branch after growth was measured, but this report posts the pre-update sample without an Witness: not run — settling the complete path requires a live GitHub 中文说明stale-base retry 会在增长测量后更新分支,但报告仍写入更新前样本,且没有 `autofix-base-updated` marker。后续轮次会把 base merge 的体积变化误归因于 PR/autofix 增长。请在成功更新 base 时写入重锚 marker,或省略该样本并强制重建基线。— gpt-5.6-sol via Qwen Code /review (v0.21.11) |
||
| # (per-workflow-run) is the dedup/order identity. | ||
| echo "<!-- autofix-growth-now src=${GROWTH_SRC:-0} test=${GROWTH_TEST:-0} over=${CRITICAL_ONLY_GROWTH:-false} round=${MARK_ROUND} run=${GITHUB_RUN_ID} key=${GROWTH_BASE_WIN:-${WINDOW:-none}} -->" | ||
|
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Critical] R6-1: Timeout, gate-rejection, abort, and no-op reports count as evidence that fixes keep growing the diff even when they changed no branch state. Failure scenario: two failed address runs push nothing but write distinct unchanged Witness: 中文说明timeout、gate-rejection、abort 与 no-op 即使没有改变分支,也会被当作“修复持续扩大 diff”的证据。两次失败运行未 push 却写入不同的 `over=true` 样本,下一轮便达到阈值并 handoff。发散判断应只使用成功产生变更的样本;无变更样本仅用于轨迹诊断。— gpt-5.6-sol via Qwen Code /review (v0.21.11) |
||
| # A sentinel ts means the agent evaluated NOTHING (crash, API | ||
| # error, gate crash) and the next scan must retry. Recording a | ||
| # judged head here would make RED_HEAD == LIVE_HEAD, so the | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.