chore: two-tier AI issue triage (draft) - #2895
Merged
Merged
Conversation
Install tier-1 classifier (on_issues_ai_triage.yaml) and tier-2 fix sweep (weekly_ai_fix.yaml) plus their helper script and prompts. Pre-merge fixes from verification: - Wait-for-ping sleep 150s -> 60s. on_issues_ping_submitter completes in 6-11s of job time across the last 10 runs; 60s covers runner-queue skew with margin. - Rule 0 rewritten to match the real ownership signal: ping_submitter posts a github-actions[bot] comment with a stable marker `<!-- addon-submitter-ping:<addon> -->`. Rule 0 now keys off that literal marker instead of fuzzy prose, and guards against @<user> == alexbelgium. - Silence one intentional shellcheck SC2016 (literal Markdown backticks) so actionlint runs clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
WalkthroughAdds AI issue-triage and add-on fix-sweep automation, including context generation, strict classification and fix prompts, scheduled/manual workflows, verdict application, and protected-path safeguards. ChangesAI issue automation
Estimated code review effort: 3 (Moderate) | ~30 minutes Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 954f1db7eb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 11
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/prompts/issue-classify.md:
- Around line 13-25: Update the ownership-matching instructions in the
issue-classification flow to require both the exact addon-submitter marker and
the comment author `github-actions[bot]` within the same comment before emitting
`owned`; preserve the existing non-`alexbelgium` submitter condition and stop
behavior.
In @.github/scripts/ai_triage_context.sh:
- Around line 65-66: Remove the `|| true` suppression from the `git
sparse-checkout set` command in the `ADDON` checkout block so a failed add-on
checkout stops triage and prevents classification without its wrapper context.
In @.github/workflows/on_issues_ai_triage.yaml:
- Around line 100-125: The label handling around LABELS must stop applying
arbitrary model-supplied labels, especially the reserved ai-triage trigger.
Build the final LABELS array from an explicit verdict-to-label allowlist,
preserve workflow-owned labels such as ai-triage only for the intended addon-bug
verdict, and filter or reject reserved labels present in verdict.json before gh
issue edit applies them.
- Around line 47-54: Update the “Checkout tooling” actions/checkout@v5 step to
set persist-credentials to false, ensuring the GitHub token is not stored in
local Git configuration before the Claude action runs. Preserve the existing
sparse-checkout configuration.
- Around line 47-64: Pin both actions in the workflow to their reviewed
immutable commit SHAs: replace the v5 reference on Checkout tooling and the v1
reference on the Classify step, using the verified SHA for each action. Preserve
the existing action configuration and secret-handling behavior.
In @.github/workflows/weekly_ai_fix.yaml:
- Around line 44-47: Update the actions/checkout configuration in the workflow
to set persist-credentials to false, preventing AI_PR_TOKEN from remaining in
local Git credentials. Preserve the existing fetch-depth and checkout token, and
keep authentication limited to the trusted post-validation push/PR step.
- Around line 44-47: Update both workflow action references, actions/checkout
and anthropics/claude-code-action, to immutable full commit SHA pins instead of
tag-style versions; retain version comments alongside the pins for tracking and
leave their existing configuration unchanged.
- Around line 29-32: Restructure the sweep workflow so the AI analysis runs in a
read-only job without repository write permissions. Move PR creation and
commenting into a separate trusted mutation job using a fine-grained short-lived
token, and retain path protections that prevent changes to .github/ and
.templates/.
- Around line 76-91: Separate the “Analyse and fix” workflow into an untrusted,
read-only analysis step and a trusted PR-creation step. Restrict the analysis
action to read-only tools, removing Write/Edit, wildcard Bash(git:*),
Bash(gh:*), and unrestricted Docker-build access; have it emit a patch or JSON
artifact. Add a trusted job that validates the artifact against the .github/,
.templates/, and config.yaml path limits before applying changes and creating
PRs.
- Around line 93-114: The Guard forbidden paths step must enforce every
hard-limit requirement from the issue-fix prompt, including protected paths,
immutable config.yaml version/upstream fields, draft-only unmerged status,
branch/add-on mapping, file and diff-size caps, and issue state. Update the PR
discovery to paginate or server-filter AI branches before applying limits, and
make any failed or incomplete validation fail closed rather than allowing the PR
through; retain the existing blocking, undo-ready, label, and comment behavior
for violations.
- Around line 57-64: Update the workflow step around the issue fetch commands to
pass the dispatch values through the step’s env configuration, then use shell
variables instead of directly interpolating inputs.issue and inputs.limit in the
run script. Validate the issue and limit values before supplying them to gh,
while preserving the existing single-issue and open ai-triage batch behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: b935f3ee-a65e-4ec9-8553-d676e4bb2853
📒 Files selected for processing (5)
.github/prompts/issue-classify.md.github/prompts/issue-fix.md.github/scripts/ai_triage_context.sh.github/workflows/on_issues_ai_triage.yaml.github/workflows/weekly_ai_fix.yaml
- Swap anthropic_api_key -> claude_code_oauth_token in both Claude steps, reading CLAUDE_CODE_OAUTH_TOKEN from the CR_PAT GitHub Environment. Both jobs now declare `environment: CR_PAT` so the environment-scoped secret is reachable. - Rename weekly_ai_fix.yaml -> daily_ai_fix.yaml (matches this repo's daily_/weekly_ filename convention) and change its cron from "0 3 * * SUN" to "0 3 * * *". - on_issues_ai_triage.yaml already triggered on issues.opened; no schedule change was needed there. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (4)
.github/workflows/daily_ai_fix.yaml (4)
48-51: 🔒 Security & Privacy | 🟠 MajorDisable checkout credential persistence.
Set
persist-credentials: false; checkout currently leaves the supplied token available for later Git operations. (github.com)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/daily_ai_fix.yaml around lines 48 - 51, Update the actions/checkout step to set persist-credentials to false alongside the existing fetch-depth and token options, preventing the supplied AI_PR_TOKEN from remaining available for subsequent Git operations.
61-67: 🔒 Security & Privacy | 🔴 CriticalStop interpolating dispatch inputs into shell code.
Pass
inputs.issueandinputs.limitthroughenv, validate them, and use shell variables. Current expression expansion allows crafted manual inputs to alter the generated script.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/daily_ai_fix.yaml around lines 61 - 67, Update the issue-fetching workflow around the issue/limit handling to stop embedding dispatch expressions directly in shell commands. Pass inputs.issue and inputs.limit through the step env, validate the resulting shell variables before use, and substitute those variables in gh issue view/list arguments while preserving the existing behavior and defaults.
97-118: 🔒 Security & Privacy | 🟠 MajorFail closed and enforce the complete fix contract.
This guard only checks
.github/and.templates/, and--limit 50is applied beforeai-fix/filtering. It can miss eligible PRs and allow other prompt-required violations; add pagination and validate all hard limits before permitting a PR.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/daily_ai_fix.yaml around lines 97 - 118, Update the “Guard forbidden paths” step to paginate through all open PRs before filtering for ai-fix/ branches, removing the effective 50-PR cap. Expand validation beyond .github/ and .templates/ to enforce every hard path and change limit required by the prompt, and fail closed by blocking, labeling, and commenting on any PR that violates them before it can proceed.
48-48: 🔒 Security & Privacy | 🟠 MajorPin third-party actions to full commit SHAs.
Replace
actions/checkout@v5andanthropics/claude-code-action@v1with verified 40-character commit SHAs. GitHub identifies full-SHA pins as the immutable release form. (docs.github.com)As per path instructions, GitHub Actions workflows require careful permissions and pinned action versions.
Also applies to: 82-82
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/daily_ai_fix.yaml at line 48, Update the workflow’s actions/checkout and anthropics/claude-code-action references to verified immutable 40-character commit SHA pins, replacing the current version tags while preserving their existing action behavior and configuration.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/daily_ai_fix.yaml:
- Line 45: Separate the Claude analysis currently in the sweep job from
repository mutation: create a read-only analysis job without Write/Edit
permissions or AI_PR_TOKEN, then pass its results to a distinct trusted job that
performs validation and PR creation using the required write permissions and
CR_PAT. Update the workflow job dependencies and permissions so the analysis job
cannot mutate the repository while preserving the existing daily AI-fix flow.
---
Duplicate comments:
In @.github/workflows/daily_ai_fix.yaml:
- Around line 48-51: Update the actions/checkout step to set persist-credentials
to false alongside the existing fetch-depth and token options, preventing the
supplied AI_PR_TOKEN from remaining available for subsequent Git operations.
- Around line 61-67: Update the issue-fetching workflow around the issue/limit
handling to stop embedding dispatch expressions directly in shell commands. Pass
inputs.issue and inputs.limit through the step env, validate the resulting shell
variables before use, and substitute those variables in gh issue view/list
arguments while preserving the existing behavior and defaults.
- Around line 97-118: Update the “Guard forbidden paths” step to paginate
through all open PRs before filtering for ai-fix/ branches, removing the
effective 50-PR cap. Expand validation beyond .github/ and .templates/ to
enforce every hard path and change limit required by the prompt, and fail closed
by blocking, labeling, and commenting on any PR that violates them before it can
proceed.
- Line 48: Update the workflow’s actions/checkout and
anthropics/claude-code-action references to verified immutable 40-character
commit SHA pins, replacing the current version tags while preserving their
existing action behavior and configuration.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 08daddc1-fb9e-49d6-b659-8b314202aacb
📒 Files selected for processing (2)
.github/workflows/daily_ai_fix.yaml.github/workflows/on_issues_ai_triage.yaml
🚧 Files skipped from review as they are similar to previous changes (1)
- .github/workflows/on_issues_ai_triage.yaml
Now that tier 2 runs daily instead of weekly, an issue left carrying ai-triage after a sweep would be re-selected and fully re-analysed the very next morning, before there's been a working day to review the first PR. Nothing previously dropped issues out of the ai-triage backlog once handled. - issue-fix.md: new hard limit 6 — relabel every issue as the last action before moving to the next one. ai:fixed / ai:upstream / ai:needs-human replace ai-triage depending on outcome. - daily_ai_fix.yaml: pre-create the three replacement labels once, up front (Claude never has to improvise a color or retry a missing-label error — wasted turns multiplied by batch size). Add a "Guard against repeat processing" step, same belt-and-braces pattern as the existing forbidden- paths guard: force-relabel to ai:needs-human anything the batch still finds carrying ai-triage after the run, independent of whether Claude's own relabeling succeeded. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified each against current code before fixing; verification details are
in the PR description update.
Fixed:
- issue-classify.md: Rule 0 now requires the addon-submitter-ping marker to
appear in a comment headed "### @github-actions[bot]", not just anywhere
in a comment or issue body, so it can't be spoofed to suppress triage.
- ai_triage_context.sh: separator-insensitive addon-slug matching (fixes
"Calibre-web" -> calibre_web, and the earlier ImmichFrame -> immich_frame
miss) before falling back to substring matching; sparse-checkout failure
now surfaces "UNRESOLVED" into the bundle instead of silently proceeding
addon-less; duplicate-issue search excludes the issue being triaged from
its own candidate list.
- on_issues_ai_triage.yaml: persist-credentials: false on the read-only
tooling checkout (nothing in that job pushes); both actions pinned to
commit SHAs (Dependabot already covers github-actions repo-wide, and
on_issues_ai.yml already sets this precedent for another AI action);
model-supplied labels are now filtered to drop anything in the ai-*/ai:*
control namespace before merging with the deterministic ai-triage/
ai:classified additions, closing a path where a verdict could
self-trigger tier 2 regardless of its actual classification.
- daily_ai_fix.yaml: both actions pinned to the same commit SHAs;
workflow_dispatch inputs.issue/inputs.limit moved out of direct
${{ }} interpolation in the run: script and into env vars with numeric
validation (template-injection); Guard forbidden paths' PR listing
limit raised 50 -> 300 so it can't silently drop ai-fix/ PRs behind
unrelated open PRs before the branch-name filter applies.
Skipped (reasons in PR description):
- persist-credentials on daily_ai_fix.yaml's checkout: disabling it
breaks the only auth path git push currently uses, and the same
AI_PR_TOKEN is already directly readable via GH_TOKEN env by that job's
unrestricted Bash(git:*)/Bash(gh:*) tools regardless.
- Splitting untrusted AI analysis into a separate job from PR-creation/
write access: legitimate defense in depth, but a full architecture
redesign, not a minimal fix.
- Full hard-limit enforcement (config.yaml immutability, diff caps,
draft-only status) replicated at the workflow level: heavy lift: the
prompt already covers these as Claude-followed instructions; only the
protected-paths check is duplicated as deterministic enforcement,
which is the single highest-severity one to enforce outside the model.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
🧹 Nitpick comments (1)
.github/workflows/daily_ai_fix.yaml (1)
58-75: 🚀 Performance & Scalability | 🔵 TrivialAdd a documented upper bound for
LIMIT_INPUT.The validation accepts any positive integer, so a manual dispatch can request a very large issue batch and feed it into the 180-minute, 300-turn Claude run. Reject values above a repository-approved maximum before calling
gh.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/daily_ai_fix.yaml around lines 58 - 75, Update the LIMIT_INPUT validation in the workflow run block to enforce the repository-approved maximum in addition to requiring a positive integer. Reject values above that maximum before either gh issue command executes, while preserving the existing error handling and accepted positive values within the bound.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In @.github/workflows/daily_ai_fix.yaml:
- Around line 58-75: Update the LIMIT_INPUT validation in the workflow run block
to enforce the repository-approved maximum in addition to requiring a positive
integer. Reject values above that maximum before either gh issue command
executes, while preserving the existing error handling and accepted positive
values within the bound.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 9866fb2e-f236-46c2-968d-6aada5ef5e9c
📒 Files selected for processing (4)
.github/prompts/issue-classify.md.github/scripts/ai_triage_context.sh.github/workflows/daily_ai_fix.yaml.github/workflows/on_issues_ai_triage.yaml
🚧 Files skipped from review as they are similar to previous changes (3)
- .github/prompts/issue-classify.md
- .github/scripts/ai_triage_context.sh
- .github/workflows/on_issues_ai_triage.yaml
This was referenced Jul 23, 2026
This was referenced Aug 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Installs the two-tier AI issue-triage system and its helpers:
.github/workflows/on_issues_ai_triage.yaml.github/workflows/weekly_ai_fix.yamlai-triageissues.github/scripts/ai_triage_context.sh.github/prompts/issue-classify.md.github/prompts/issue-fix.mdKept in DRY_RUN (tier 1) and draft-PR-only (tier 2). Requires secrets
ANTHROPIC_API_KEYandAI_PR_TOKENbefore it does anything live.Pre-merge verification
1. Rule 0 vs.
on_issues_ping_submitter.yml— assumption confirmed, format tightenedping_submitterdoes post a comment naming a responsible person, so Rule 0'spremise holds (it is not a label or an assignee). The comment is authored by
github-actions[bot]and has a stable, greppable shape:The hidden marker survives into the context bundle verbatim (the bundle dumps
.comments[].body), so the classifier can match it exactly. Rule 0 rewrittento key off the literal
<!-- addon-submitter-ping:marker instead of inferringownership from prose, and to guard the
@<user> == alexbelgiumedge.Sleep adjusted to real runtime. The two workflows race off the same
issues.openedevent. Across the last 10ping_submitterruns, job time was6–11 s (e.g.
06:23:12→06:23:21,07:21:33→07:21:44). Changedsleep 150→sleep 60, which still leaves generous margin for runner-queueskew between the two jobs.
2.
ai_triage_context.shslug resolution — tested on real issues, failures reported (not loosened)Per instruction, the match was not loosened — here are the failures found by
running the script's exact resolution against real issue titles:
🐛 [Immich Frame] …immich_frame🐛 [Changedetection.io] …changedetection.io🐛 [BattybirdNET-Pi Docker Standalone] …battybirdnet-pi(via substring fallback)[Filebrowser Quantum] …(canonical)filebrowser_quantum🐛 [ImmichFrame] …(no space)immich✨ [REQUEST] BattyBirdnet-Pi …[REQUEST], addon name ignoredFileBrowser Quantum: …(no brackets)Three real-world failure classes:
[ImmichFrame]has noseparator, so no directory matches exactly and the substring fallback
(
index(candidate, dirname)) latches onto the shorterimmich. Thissilently points the tier-2 fix bot at the wrong add-on's source. Real hits:
🐛 [ImmichFrame] Config options no longer work #2586, ✨ [REQUEST] ImmichFrame - Multi Account support #2582.
head -n1on the first[...]grabs[REQUEST]/[Bug]:/[Addon name]and never sees the real name (✨ [REQUEST] BattyBirdnet-Pi x86-64 #1498, [Bug]: Immich add-on v3.0.1-2: microservices worker crash-loops on startup, DB init race condition #2810, 🐛 [Addon name] birdnet-go #2300,🐛 [Addon name] Birdnet update #2678, 🐛 [Addon name] birdnet-pi-amd64:2026.03.01 #2839, ❓ [addon] immich update #2178).
FileBrowser Quantum: …, FileBrowser Quantum: Configuration doesn't survive over restarts #2752) → unresolved.Cases 1–2 are the ones to weigh: an
UNRESOLVEDbundle is caught by theclassifier's
low-confidence rule, but a confident wrong slug (#2586) is not.Left as-is per the "report, don't silently loosen" instruction — flagging for a
decision on the fallback heuristic.
3. actionlint + yamllint
was one
info-levelSC2016on intentional literal-Markdown backticks;silenced with a scoped
# shellcheck disable=SC2016..yamllintconfig and itslint.ymlsuper-linter explicitly excludes
.github/workflows/.*, so workflows are notlinted in CI. Under stock yamllint defaults the only items are the universal
truthyfalse-positive on the requiredon:key andline-length>80onshell/comment lines — neither enforced here, so left unchanged rather than
reformatting working shell.
ai_triage_context.shis shellcheck-clean (super-linter does lint.github/scripts/).Note:
anthropics/claude-code-action(requested evaluation)The action is already correctly integrated in both workflows, so no change
was needed to "include" it:
@v1pinned;anthropic_api_keyis a valid auth input.prompt, which puts the action in unattendedautomation mode (no
@claudemention required) — correct forevent/cron-driven triage.
claude_argsflag verified against theclaudeCLI:--model(
claude-sonnet-5,claude-opus-4-8),--effort(low/xhigh),--max-turns,--allowedToolsare all real flags.No structural improvement warranted; the integration follows the documented
pattern.
🤖 Generated with Claude Code
Summary by CodeRabbit