feat: restore Agent Shin on safe triggers with staged rollout mechanics

[mateo-berri]

Relevant issues

Follow-up to #30784, which removed these two workflows because they ran on pull_request_target (see also #30433, where they were introduced)

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

zizmor audit of the three Agent Shin workflows at commit 13f3462, confirming the dangerous trigger stays gone:

$ uvx zizmor --offline .github/workflows/triage_pr_with_llm.yml .github/workflows/review_gate.yml .github/workflows/triage_reconsider.yml
 INFO zizmor: zizmor v1.26.1
 INFO audit: zizmor: completed .github/workflows/review_gate.yml
 INFO audit: zizmor: completed .github/workflows/triage_pr_with_llm.yml
 INFO audit: zizmor: completed .github/workflows/triage_reconsider.yml
No findings to report. Good job! (27 suppressed)

Workflow files are only dispatchable once they exist on the default branch, so the live end-to-end proof runs after merge. Reviewer steps, all dry-run and side-effect free while AGENT_SHIN_ENABLED is unset:

1. gh workflow run &#34;Agent Shin — PR triage&#34; -f pr_number=<open external PR> -f close=false
2. gh workflow run &#34;Agent Shin — review gate&#34; -f pr_number=<same PR> -f close=false
3. Open each run in the Actions tab and check the step summary contains the judge's verdict for that PR, with a ::notice:: line confirming dry-run mode and no comment/label/close on the PR itself
4. Set repo variable AGENT_SHIN_MODE=lite, re-run step 2, and check the step summary shows would-notify-lite (the one-time 7-day notice preview) instead of would-close for a failing PR; unset the variable afterwards
5. Comment @agent-shin reconsider on a bot-closed PR that has no Greptile score and check the run summary shows would-reconsider-needs-greptile pointing the author at @greptileai

Type

🚄 Infrastructure

Changes

Restores triage_pr_with_llm.yml and review_gate.yml, which #30784 deleted because they ran on pull_request_target. The bodies are the same as the deleted versions except for the trigger surface and the lines that depended on it

triage_pr_with_llm.yml now runs only on workflow_dispatch. review_gate.yml runs on workflow_dispatch plus its original daily 09:30 UTC schedule sweep, which #30784 also deleted and which this PR brings back. Neither workflow can be fired by a fork author: workflow_dispatch requires actions:write, so only maintainers and the agent-shin bridge app can trigger them. The pull_request_target-specific plumbing (EVENT_PR, the force-dry-run branch for automatic PR events, and the conditional OPENAI_API_KEY expression keyed on event name) is removed, with the close/dry-run gating on AGENT_SHIN_ENABLED kept identical

Instant reaction to fork PR events, the one thing pull_request_target provided that these triggers cannot, is restored by a separate Cloudflare Worker (private repo agent-shin). It receives GitHub App pull_request webhooks, verifies the HMAC signature, and dispatches these two workflows with the PR number. It refuses to dispatch unless the repo variable AGENT_SHIN_ENABLED is exactly true, preserving the old guarantee that an external user cannot force paid LLM calls by churning fork PRs while the bot is in dry-run. The bridge's GitHub App carries only actions:write, variables:read, and metadata:read, so the privileged triage logic keeps running exclusively as base-repo code inside Actions

On top of the restore, this PR adds the mechanics for the staged public rollout:

The review gate now maintains a label pair instead of a single label: ready for review on pass, not ready on fail, reconciled idempotently on every run so the pair always shows the current verdict

Lite mode: while repo variable AGENT_SHIN_MODE is the exact string lite, the review gate never closes anything and instead posts a one-time notice that closes start in 7 days, linking the policy blog post from AGENT_SHIN_POLICY_URL and deduped via a hidden HTML marker. Labels still reconcile and regression comments quote the 7-day window instead of the 24h grace. Any other value of the variable, including unset, gives full behavior, mirroring the fail-safe exact-string convention used for AGENT_SHIN_ENABLED

Reconsider on a closed PR is now held to the full reopen bar: the PR must carry a Greptile confidence score of at least 4/5 (a missing score fails fast, before any LLM spend, with a comment walking the author through @greptileai and the QA evidence requirements), and the linked-issue short-circuit is bypassed so the rubric's end-to-end QA evidence check always applies. A pass reopens the PR and swaps the label pair to ready for review. Reconsider on an open PR now runs the review gate for that PR, so an author who just fixed their description gets the tag flipped immediately instead of waiting for the daily sweep

The PR rubric is tightened to match the rollout policy: bug-fix proof must show the failure before the change and the fix after, and command-based proof produced by a custom script only counts when the script source is visible in the body (a collapsible section is fine) or a linked gist

The default triage model moves to gpt-5.6-luna

Tests: 247 passing across the Agent Shin suites, including new coverage for the label pair lifecycle, every lite-mode branch (one-time notice, dedup, dry-run, regression wording, never-closes-even-past-grace), the Greptile reconsider gate (missing score, low score, dry-run, PR-only), the linked-issue bypass in reconsider mode, the model default, and static workflow guardrails pinning the exact-string lite comparison and the open-PR-to-review-gate routing

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

This PR restores two Agent Shin workflows (triage_pr_with_llm.yml, review_gate.yml) that were previously deleted due to unsafe pull_request_target triggers, now running exclusively on workflow_dispatch and a daily schedule so fork authors cannot fire them. It also adds staged-rollout mechanics on top of the restore.

• Trigger hardening: both new workflows require actions:write to dispatch; the review_gate.yml schedule sweep and the triage_reconsider.yml open-PR path route to --review-gate (not --reconsider) so the label pair flips immediately on @agent-shin reconsider.
• Label pair: ready for review / not ready are now reconciled idempotently on every run; remove_label suppresses 404s so unconditional cleanup in the reopen path is safe.
• Lite mode: AGENT_SHIN_MODE=lite (exact-string match) disables closes and posts a one-time 7-day notice instead, deduped via LITE_NOTICE_MARKER; labels still reconcile.
• Greptile gate on reconsider: closed-PR reconsiders now require a Greptile confidence score ≥ 4/5 before any LLM spend; missing or low scores post a recovery comment with the rate-limit marker to prevent budget abuse.

Confidence Score: 5/5

Safe to merge. All destructive paths are gated behind exact-string env-var checks, no pull_request_target trigger exists, and remove_label handles absent labels gracefully.

The trigger surface change is the most sensitive part of this PR and was handled correctly: both new workflows use workflow_dispatch (requiring actions:write) with no pull_request_target. The label-pair state machine is idempotent and well-tested. The Greptile gate on reconsider runs before any LLM call, preventing budget abuse. 247 tests cover every new branch.

No files require special attention.

Important Files Changed

Filename	Overview
.github/scripts/triage_with_llm.py	Core triage script extended with Greptile-score gate, NOT_READY_LABEL pair reconciliation, lite mode with LITE_NOTICE_MARKER dedup, and linked-issue bypass disabled in reconsider mode.
.github/workflows/review_gate.yml	New workflow using workflow_dispatch + daily schedule. SHA-pinned actions, exact-string checks for AGENT_SHIN_ENABLED and AGENT_SHIN_MODE.
.github/workflows/triage_pr_with_llm.yml	New workflow using workflow_dispatch only. Requires AGENT_SHIN_ENABLED=true AND DISPATCH_CLOSE=true for the destructive path.
.github/workflows/triage_reconsider.yml	Routes open-PR reconsiders to --review-gate instead of --reconsider. Lite mode and policy URL env vars wired in.
tests/test_litellm/test_github_review_gate.py	Added label-pair lifecycle tests and full lite-mode branch coverage.
tests/test_litellm/test_github_triage_with_llm.py	Added Greptile-gate tests, rewrote linked-issue reconsider test to verify bypass is disabled, added label-pair assertions.
tests/test_litellm/test_github_triage_workflows.py	Added static guardrail tests for open-PR reconsider routing and exact-string lite-mode checks.

_{Reviews (3): Last reviewed commit: "feat: staged rollout mechanics for Agent..." | Re-trigger Greptile}

[mateo-berri]

@greptileai

---

Generated by Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing claude/thread-summary-lezdmb (13f3462) with litellm_internal_staging (3e9e520)}

[mateo-berri]

@greptileai

---

Generated by Claude Code

[devin-ai-integration]

🚅 Hi, thanks for the PR! I'm Agent Shin, the automated triage bot for this repository. What's this and why am I getting it?

I read the description against our contribution rubric. Here's how it lined up:

What you got right:

• ✅ Linked a related GitHub issue
• ✅ Clear problem description
• ✅ Expected vs. actual behavior
• ✅ End-to-end QA proof

What's still missing:

• (see explanation below)

The PR includes a clear problem description, expected-vs-actual behavior, and command/output evidence via the zizmor audit plus described workflow verification steps. It also links a related issue, so both triage requirements are satisfied.

If the description isn't updated in the next 24 hours, I'll auto-close this PR. That's not us saying we don't care about the change; we want the open-PR list to mirror what a maintainer can act on right now, so contributors don't get lost in a backlog. A closed PR is a soft "park this for later," not a rejection. Take your time; everything below still works after the close.

During the grace period: just update the PR description with the missing pieces. No need to ping me; I'll re-check on the next sweep and skip the auto-close if it now passes. See what counts as QA proof for the full rubric (a linked issue alone isn't enough; it covers context, not proof).

If the PR does get auto-closed in 24 hours, you still have easy recovery paths:

• Comment @agent-shin reconsider after updating the description. I'll re-evaluate and reopen the PR if it now passes.
• Comment @greptileai to request a fresh Greptile review; that still works even after the PR is closed, and a stronger score is one of the signals that lifts the PR back into the queue. So a low Greptile score isn't a blocker either.

Internal BerriAI contributors: this rubric doesn't apply to you; ping a maintainer.

(I'm an LLM, so I'm not infallible. If you think I got this wrong, ping a maintainer; they'll override me.)