From 62c49aeebf60da6f88978cfbd505cb3276727c94 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 21 Jul 2026 00:31:02 +0000 Subject: [PATCH 1/2] ci: fail AI triage loudly when the session bails before posting (#338) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A Claude Code CLI update made Task sub-agents background-by-default, so the triage session fanned out its search agents, said it would wait for them, and ended its turn — which terminates the headless session with no comment posted. The result record is a clean success, so the is_error watchdog stayed green (a new variant of the #317 silent-no-op shape). - Command files + workflow prompt: Task calls must pass run_in_background: false, and the session must never end its turn while a sub-agent is pending. - Workflow prompt now requires a machine-checkable final message (`TRIAGE-RESULT: commented|refreshed|skipped ()`); the watchdog jq additionally requires the final result text to be nothing but such sentinel lines, so any future mid-task bailout fails the job instead of showing green. - Search agent rules: one prefix-allowlisted gh command per Bash call (loops/echo prefixes were being permission-denied) and a 6-search cap per agent (bursts were 403ing on the shared API rate limit). Fixes #338 Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01K1LrTrVRbtdh8paN7f67CB --- .claude/commands/triage-dedupe.md | 16 ++++- .claude/commands/triage-find-duplicate-prs.md | 17 ++++- .claude/commands/triage-find-issues.md | 17 ++++- .github/workflows/ai-triage.yml | 65 +++++++++++++++---- 4 files changed, 99 insertions(+), 16 deletions(-) diff --git a/.claude/commands/triage-dedupe.md b/.claude/commands/triage-dedupe.md index 7ae937957..4ef094d8e 100644 --- a/.claude/commands/triage-dedupe.md +++ b/.claude/commands/triage-dedupe.md @@ -32,7 +32,16 @@ run locally), and `AUTOCLOSE` (`active` or `off`; assume `off` locally). If Summarize the issue first: symptoms, component/file names, API/prop names, exact error strings, and the general area. Then fan out FIVE Task agents in -a single message, each pursuing ONE distinct strategy: +a single message, each pursuing ONE distinct strategy. + +Sub-agents run SYNCHRONOUSLY: every Task call MUST pass +`run_in_background: false`. If the runtime backgrounds them anyway, NEVER +end your turn while any sub-agent is still pending — in the headless CI +session an ended turn terminates the session immediately, orphaning the +agents before any comment is posted (#338). Collect every agent's result, +then continue with the filter pass. + +The five strategies: 1. **Error strings** — exact quoted messages, stack-trace lines, warning text. 2. **Component + file names** — `Button`, `Modal`, `useConfig`, source paths. @@ -46,6 +55,11 @@ Rules for every agent: the `--repo REPO` flag). Never search outside it. - Use `gh search issues`, `gh issue list --search`; include closed issues (a duplicate of a closed issue is still a duplicate). +- ONE `gh` command per Bash call, starting with the `gh` binary — no shell + loops, `echo` prefixes, or `;`/`&&` chains (the permission allowlist + matches command prefixes; anything else is denied and wastes turns). +- At most 6 searches per agent — the GitHub API limit is shared across all + agents, and a burst of searches 403s the whole session. - Return candidate numbers with title and a one-line justification each. ## Filter pass (you, not the agents) diff --git a/.claude/commands/triage-find-duplicate-prs.md b/.claude/commands/triage-find-duplicate-prs.md index 04e90f4d2..2682f8c09 100644 --- a/.claude/commands/triage-find-duplicate-prs.md +++ b/.claude/commands/triage-find-duplicate-prs.md @@ -25,7 +25,16 @@ assume `labeled` locally). If `NUMBER` is missing, ask. ## Search — 3 parallel agents Read the PR title, body, and changed file list, then fan out THREE Task -agents in a single message, one strategy each: +agents in a single message, one strategy each. + +Sub-agents run SYNCHRONOUSLY: every Task call MUST pass +`run_in_background: false`. If the runtime backgrounds them anyway, NEVER +end your turn while any sub-agent is still pending — in the headless CI +session an ended turn terminates the session immediately, orphaning the +agents before any comment is posted (#338). Collect every agent's result, +then continue with the filter pass. + +The three strategies: 1. **Changed files** — other PRs touching the same components or paths. 2. **Title + body keywords** — the strongest words plus synonyms. @@ -33,7 +42,11 @@ agents in a single message, one strategy each: Rules for every agent: EVERY search scoped `repo:REPO is:pr is:open is:unmerged` (via `gh search prs` or -`gh pr list --search`); exclude the target PR itself and draft PRs; return +`gh pr list --search`); exclude the target PR itself and draft PRs; ONE +`gh` command per Bash call, starting with the `gh` binary — no shell +loops, `echo` prefixes, or `;`/`&&` chains (the permission allowlist +matches command prefixes; anything else is denied and wastes turns); at +most 6 searches per agent (the shared API limit 403s on bursts); return numbers + titles + a one-line justification. ## Filter pass diff --git a/.claude/commands/triage-find-issues.md b/.claude/commands/triage-find-issues.md index 1b212c5f0..01cf070e2 100644 --- a/.claude/commands/triage-find-issues.md +++ b/.claude/commands/triage-find-issues.md @@ -28,7 +28,16 @@ EXACTLY ONE comment (or nothing). Context from the caller: `REPO`, `NUMBER` Read the PR title, body, and `gh pr diff NUMBER --repo REPO` (never check out or execute its code). Extract signals, then fan out FIVE Task agents in -a single message, one strategy each: +a single message, one strategy each. + +Sub-agents run SYNCHRONOUSLY: every Task call MUST pass +`run_in_background: false`. If the runtime backgrounds them anyway, NEVER +end your turn while any sub-agent is still pending — in the headless CI +session an ended turn terminates the session immediately, orphaning the +agents before any comment is posted (#338). Collect every agent's result, +then continue with the filter pass. + +The five strategies: 1. **Error strings** — messages or test names the diff fixes or touches. 2. **Component + file names** — components and paths the diff changes. @@ -38,7 +47,11 @@ a single message, one strategy each: Rules for every agent: scope EVERY search to this repository (`repo:REPO` or `--repo REPO`); OPEN issues only (`is:open` / -`--state open`); return numbers + titles + a one-line justification. +`--state open`); ONE `gh` command per Bash call, starting with the `gh` +binary — no shell loops, `echo` prefixes, or `;`/`&&` chains (the +permission allowlist matches command prefixes; anything else is denied and +wastes turns); at most 6 searches per agent (the shared API limit 403s on +bursts); return numbers + titles + a one-line justification. ## Filter pass diff --git a/.github/workflows/ai-triage.yml b/.github/workflows/ai-triage.yml index a960c0503..26c66f88d 100644 --- a/.github/workflows/ai-triage.yml +++ b/.github/workflows/ai-triage.yml @@ -63,6 +63,17 @@ name: AI Triage # file-editing tools, because the session ingests untrusted issue/PR # text. # +# SILENT NO-OPS (#317, #338): the action reports step success even when the +# session did no useful work, so the "Fail on silent session error" step +# checks the execution file. Two known shapes: an is_error result record +# (#317 — API/auth failure on the first call), and a CLEAN result record +# from a session that ended its turn while background Task sub-agents were +# still running (#338 — a CLI update made Task background-by-default; an +# ended turn kills the headless session and orphans the agents). The prompt +# therefore forbids backgrounded sub-agents AND requires a final +# `TRIAGE-RESULT:` sentinel line per command; the watchdog fails the job +# unless the result text is nothing but such lines. +# # KILL SWITCHES: AI_TRIAGE_MODE=off (this workflow only) or # AI_LOOP_ENABLED=false (all Claude spend, repo-wide). # @@ -325,6 +336,24 @@ jobs: commands and do NOT use the SlashCommand tool — it is not permitted in this session and every attempt will be denied. + Sub-agents run SYNCHRONOUSLY (#338): every Task call MUST pass + run_in_background: false. If the runtime backgrounds them + anyway, you MUST NOT end your turn while any sub-agent is still + pending — this is a headless one-shot session, and an ended + turn terminates it immediately, orphaning the agents with no + comment posted. Keep collecting results until every sub-agent + has returned, then finish the command's filter/post steps. + + FINAL MESSAGE (machine-checked; the job FAILS without it): your + last output message must consist of exactly one line per + command you ran, each of the form + TRIAGE-RESULT: commented | refreshed | skipped () + e.g. `TRIAGE-RESULT: triage-dedupe commented` or + `TRIAGE-RESULT: triage-find-duplicate-prs skipped (no credible + duplicates)` — and nothing else. A command's silent exit is + still reported here as `skipped ()`; "silent" means no + comment on the item, not no sentinel. + Idempotency (the commands restate this; it is binding): - TRIGGER is opened: if a command's marker comment already exists on the item, that command exits silently — post @@ -344,12 +373,19 @@ jobs: post nothing beyond what the commands specify. # The action reports step success even when the Claude session itself - # errors (result message has is_error: true — e.g. an API auth or - # usage-limit failure on the very first call). The item then silently - # gets no triage comment while the run shows green — the exact #317 - # failure shape. Fail loudly instead. An empty/missing execution file - # (e.g. the action's workflow-validation skip) is tolerated: there was - # no session to judge. + # did no useful work. Two known silent-no-op shapes (see header): + # #317 — the result record has is_error: true (e.g. an API auth or + # usage-limit failure on the very first call). + # #338 — the result record is a CLEAN success, but the session + # ended its turn while background Task sub-agents were still + # running ("I'll wait for them to finish…"), so nothing was + # ever posted. Session status alone cannot catch this, which + # is why the prompt demands the TRIAGE-RESULT sentinel: a + # session that actually finished its commands ends with only + # sentinel lines; any other final text fails here. + # Fail loudly on both. An empty/missing execution file (e.g. the + # action's workflow-validation skip) is tolerated: there was no + # session to judge. - name: Fail on silent session error if: steps.claude.outcome == 'success' env: @@ -361,15 +397,22 @@ jobs: exit 0 fi # Fail closed: accept a JSON array or NDJSON, and require a final - # result record whose is_error is exactly false. Malformed JSON, - # a missing result record, or any other shape fails the job. + # result record whose is_error is exactly false AND whose result + # text is nothing but TRIAGE-RESULT sentinel lines (at least + # one). Malformed JSON, a missing result record, a mid-task + # bailout message, or any other shape fails the job. if jq -es '[.[] | if type == "array" then .[] else . end] | map(select(type == "object" and .type == "result")) - | last | .is_error == false' \ + | last + | (.is_error == false) + and ((.result // "") | split("\n") + | map(select(length > 0)) + | (length > 0) + and all(startswith("TRIAGE-RESULT:")))' \ "$EXEC_FILE" >/dev/null 2>&1; then - echo "session result is_error: false — triage session completed" + echo "session ended cleanly with TRIAGE-RESULT sentinel(s) — triage completed" else - echo "::error::Claude triage session did not complete cleanly (is_error true, missing result record, or unparsable execution file) — no triage comment was posted. See the session output above." + echo "::error::Claude triage session did not complete its commands (is_error true, missing/non-sentinel final message, or unparsable execution file — see #317/#338). The item likely got no triage comment. See the session output above." exit 1 fi From d2ffb51917aa965dfa525b3f64a24e4fb1495a82 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 21 Jul 2026 01:28:15 +0000 Subject: [PATCH 2/2] ci: enforce per-item sentinel count in triage watchdog; ban pipes in agent rules MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Deep-review follow-up on #340: the watchdog accepted any nonzero number of TRIAGE-RESULT lines, so a PR run that completed triage-find-issues but bailed during triage-find-duplicate-prs passed on one sentinel. The gate step's is_pr output now feeds the check, which requires exactly one sentinel for issues and two for PRs. The search-agent command rules also now name pipes and command substitution alongside loops/chains — the same prefix allowlist denies those too. Refs #338 Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01K1LrTrVRbtdh8paN7f67CB --- .claude/commands/triage-dedupe.md | 5 ++-- .claude/commands/triage-find-duplicate-prs.md | 9 +++--- .claude/commands/triage-find-issues.md | 9 +++--- .github/workflows/ai-triage.yml | 29 +++++++++++++------ 4 files changed, 33 insertions(+), 19 deletions(-) diff --git a/.claude/commands/triage-dedupe.md b/.claude/commands/triage-dedupe.md index 4ef094d8e..0d0914228 100644 --- a/.claude/commands/triage-dedupe.md +++ b/.claude/commands/triage-dedupe.md @@ -56,8 +56,9 @@ Rules for every agent: - Use `gh search issues`, `gh issue list --search`; include closed issues (a duplicate of a closed issue is still a duplicate). - ONE `gh` command per Bash call, starting with the `gh` binary — no shell - loops, `echo` prefixes, or `;`/`&&` chains (the permission allowlist - matches command prefixes; anything else is denied and wastes turns). + loops, `echo` prefixes, `;`/`&&` chains, pipes (`| head`, `| jq`), or + command substitution (`$(...)`) (the permission allowlist matches command + prefixes; anything else is denied and wastes turns). - At most 6 searches per agent — the GitHub API limit is shared across all agents, and a burst of searches 403s the whole session. - Return candidate numbers with title and a one-line justification each. diff --git a/.claude/commands/triage-find-duplicate-prs.md b/.claude/commands/triage-find-duplicate-prs.md index 2682f8c09..4cf884eba 100644 --- a/.claude/commands/triage-find-duplicate-prs.md +++ b/.claude/commands/triage-find-duplicate-prs.md @@ -44,10 +44,11 @@ Rules for every agent: EVERY search scoped `repo:REPO is:pr is:open is:unmerged` (via `gh search prs` or `gh pr list --search`); exclude the target PR itself and draft PRs; ONE `gh` command per Bash call, starting with the `gh` binary — no shell -loops, `echo` prefixes, or `;`/`&&` chains (the permission allowlist -matches command prefixes; anything else is denied and wastes turns); at -most 6 searches per agent (the shared API limit 403s on bursts); return -numbers + titles + a one-line justification. +loops, `echo` prefixes, `;`/`&&` chains, pipes, or command substitution +(the permission allowlist matches command prefixes; anything else is +denied and wastes turns); at most 6 searches per agent (the shared API +limit 403s on bursts); return numbers + titles + a one-line +justification. ## Filter pass diff --git a/.claude/commands/triage-find-issues.md b/.claude/commands/triage-find-issues.md index 01cf070e2..8f285d8ce 100644 --- a/.claude/commands/triage-find-issues.md +++ b/.claude/commands/triage-find-issues.md @@ -48,10 +48,11 @@ The five strategies: Rules for every agent: scope EVERY search to this repository (`repo:REPO` or `--repo REPO`); OPEN issues only (`is:open` / `--state open`); ONE `gh` command per Bash call, starting with the `gh` -binary — no shell loops, `echo` prefixes, or `;`/`&&` chains (the -permission allowlist matches command prefixes; anything else is denied and -wastes turns); at most 6 searches per agent (the shared API limit 403s on -bursts); return numbers + titles + a one-line justification. +binary — no shell loops, `echo` prefixes, `;`/`&&` chains, pipes, or +command substitution (the permission allowlist matches command prefixes; +anything else is denied and wastes turns); at most 6 searches per agent +(the shared API limit 403s on bursts); return numbers + titles + a +one-line justification. ## Filter pass diff --git a/.github/workflows/ai-triage.yml b/.github/workflows/ai-triage.yml index 26c66f88d..c5f1ba5bb 100644 --- a/.github/workflows/ai-triage.yml +++ b/.github/workflows/ai-triage.yml @@ -72,7 +72,9 @@ name: AI Triage # ended turn kills the headless session and orphans the agents). The prompt # therefore forbids backgrounded sub-agents AND requires a final # `TRIAGE-RESULT:` sentinel line per command; the watchdog fails the job -# unless the result text is nothing but such lines. +# unless the result text is nothing but such lines, exactly one per +# expected command (one for issues, two for PRs — so finishing the first +# PR command and bailing during the second still fails). # # KILL SWITCHES: AI_TRIAGE_MODE=off (this workflow only) or # AI_LOOP_ENABLED=false (all Claude spend, repo-wide). @@ -346,7 +348,10 @@ jobs: FINAL MESSAGE (machine-checked; the job FAILS without it): your last output message must consist of exactly one line per - command you ran, each of the form + command listed above — ONE line when IS_PR is false, TWO lines + when IS_PR is true (the job verifies the count, so a run that + finishes the first command but bails during the second cannot + pass) — each of the form TRIAGE-RESULT: commented | refreshed | skipped () e.g. `TRIAGE-RESULT: triage-dedupe commented` or `TRIAGE-RESULT: triage-find-duplicate-prs skipped (no credible @@ -390,6 +395,7 @@ jobs: if: steps.claude.outcome == 'success' env: EXEC_FILE: ${{ steps.claude.outputs.execution_file }} + IS_PR: ${{ steps.gate.outputs.is_pr }} run: | set -euo pipefail if [ -z "$EXEC_FILE" ] || [ ! -f "$EXEC_FILE" ]; then @@ -398,21 +404,26 @@ jobs: fi # Fail closed: accept a JSON array or NDJSON, and require a final # result record whose is_error is exactly false AND whose result - # text is nothing but TRIAGE-RESULT sentinel lines (at least - # one). Malformed JSON, a missing result record, a mid-task - # bailout message, or any other shape fails the job. - if jq -es '[.[] | if type == "array" then .[] else . end] + # text is nothing but TRIAGE-RESULT sentinel lines — exactly one + # per expected command (issues run one command, PRs run two), so + # a session that finishes the first PR command but bails during + # the second cannot pass on the strength of one sentinel. + # Malformed JSON, a missing result record, a mid-task bailout + # message, or any other shape fails the job. + if [ "$IS_PR" = "true" ]; then EXPECTED=2; else EXPECTED=1; fi + if jq -es --argjson n "$EXPECTED" \ + '[.[] | if type == "array" then .[] else . end] | map(select(type == "object" and .type == "result")) | last | (.is_error == false) and ((.result // "") | split("\n") | map(select(length > 0)) - | (length > 0) + | (length == $n) and all(startswith("TRIAGE-RESULT:")))' \ "$EXEC_FILE" >/dev/null 2>&1; then - echo "session ended cleanly with TRIAGE-RESULT sentinel(s) — triage completed" + echo "session ended cleanly with $EXPECTED TRIAGE-RESULT sentinel(s) — triage completed" else - echo "::error::Claude triage session did not complete its commands (is_error true, missing/non-sentinel final message, or unparsable execution file — see #317/#338). The item likely got no triage comment. See the session output above." + echo "::error::Claude triage session did not complete its commands (is_error true, wrong sentinel count for the item type, non-sentinel final message, or unparsable execution file — see #317/#338). The item likely got no (or incomplete) triage. See the session output above." exit 1 fi