feat(triage): Agent Shin — LLM-judge + Greptile auto-close (any age, any draft state) + @agent-shin reconsider flow

[mateo-berri]

Relevant issues

Implements the OSS-triage idea discussed in the #random Slack thread on 4/30 and expanded on 5/17:

• "close PRs which are 1wk+ and below Greptile 4/5 to clean up the backlog" (Krrish)
• "Agent Shin can be the first line of triage… auto-close issues/PRs that don't fit our rubric with a comment explaining why, and re-evaluate on reopen" (Mateo, Ryan)
• Run only on external OSS contributors — exempt internal BerriAI employees
• Don't actually start closing yet — ship in dry-run mode so the team can QA the bot's verdicts before flipping it on

Updated per 5/17 follow-up

• Auto-close every PR, regardless of age and draft status (the open-PR queue should equal the queue internal collaborators need to action on).
• OSS contributors cannot reopen a bot-closed PR (long-standing GitHub limitation), so close-comments now point them at "open a new PR" or @agent-shin reconsider instead of "reopen the PR — I'll re-evaluate". A new issue_comment workflow handles @agent-shin reconsider end-to-end.

What this PR ships

1. Updated contribution templates (.github/)

• pull_request_template.md — new rubric at the top: a PR passes if it either (A) links a related GitHub issue (Fixes #1234, …) or (B) provides a clear problem description + expected vs. actual + visual QA proof. Updated wording: every external PR is triaged regardless of draft status / age, and the "what to do if auto-closed" section recommends opening a new PR or commenting @agent-shin reconsider. Linear-ticket section preserved for internal contributors.
• ISSUE_TEMPLATE/bug_report.yml — splits "what happened" into separate Actual / Expected fields; makes log/screenshot a required textarea; warning banner about auto-triage.
• ISSUE_TEMPLATE/feature_request.yml — requires concrete motivation + example, not just a one-liner; same warning banner.

2. Agent Shin (LLM-as-judge) — .github/scripts/triage_with_llm.py

Single entrypoint that triages one PR or issue per invocation. Decision flow:

Condition	Outcome
state != open (regular mode)	skip-not-open
state != closed (reconsider mode)	skip-not-closed
author_association ∈ {OWNER, MEMBER, COLLABORATOR} or login ends [bot]	skip-internal-author
PR body matches Fixes/Closes/Resolves #N or full issue URL	pass-linked-issue (no LLM call) — in reconsider mode also reopens
LLM judge returns verdict=pass	pass-llm (regular) / reopened (reconsider)
LLM judge returns verdict=fail, --close not set	would-close (dry-run)
LLM judge returns verdict=fail, --close set	closed (comment posted + state=closed)
LLM judge returns verdict=fail in reconsider mode	reconsider-still-failing (posts "still missing X" comment, leaves closed)
LLM call or JSON parse fails	skip-llm-error (never destructive)

The judge uses an OpenAI-compatible chat endpoint (OPENAI_API_KEY secret, optional OPENAI_BASE_URL/TRIAGE_MODEL overrides; defaults to gpt-5.4-mini). Prompts strip HTML comments from the body so empty-template-placeholder PRs are correctly judged as empty.

3. Greptile-score auto-closer — .github/scripts/close_low_quality_prs.py

• No more age filter by default. --min-age-days defaults to 0; the daily scheduled sweep closes any external-author PR (including drafts) the moment Greptile scores it <4/5. The flag is preserved as an opt-in safety net for one-off backfill runs that want to spare very-young PRs.
• No more draft skip. A draft Greptile scored 2/5 is still in the queue internal collaborators have to action on, so it's eligible to close. Authors who genuinely need a long-lived draft can attach the wip opt-out label (unchanged).
• Updated close-comment wording to recommend "open a new PR" + @agent-shin reconsider instead of "reopen the PR" (OSS contributors can't reopen a bot-closed PR).

4. New @agent-shin reconsider workflow — .github/workflows/triage_reconsider.yml

issue_comment-triggered. When the PR/issue author OR an internal collaborator comments @agent-shin reconsider on a CLOSED PR/issue, the bot re-runs LLM-judge triage on the current title+body and:

• on PASS → posts a "re-evaluated and reopened" comment + reopens via the bot's GH_TOKEN write access (which the OSS author themselves does NOT have).
• on FAIL → posts a "still missing X" comment and leaves the PR closed so the contributor can iterate again.

Authorization is gated via a step output: only the PR/issue author (comment.user.login == issue.user.login) or a commenter whose author_association is OWNER/MEMBER/COLLABORATOR proceeds past the auth step. Random commenters never reach the destructive steps. Globally gated on vars.AGENT_SHIN_ENABLED == 'true' (positive form, matching the existing fail-safe pattern).

5. Workflows (dry-run by default)

• .github/workflows/triage_pr_with_llm.yml — pull_request_target: [opened, reopened] + manual workflow_dispatch. Fires on draft and ready-for-review PRs alike (GitHub's pull_request_target includes drafts).
• .github/workflows/triage_issue_with_llm.yml — issues: [opened, reopened] + manual workflow_dispatch.
• .github/workflows/close_low_quality_prs.yml — daily scheduled Greptile-score sweep; default age filter is now 0.
• .github/workflows/triage_reconsider.yml — issue_comment trigger for the new reconsider flow.

All four are gated on a single repo variable AGENT_SHIN_ENABLED. Until you set it to "true" in Settings → Secrets and variables → Actions → Variables, every run writes its verdict to the workflow step summary only — no public comments, no closures, no reopens.

6. Tests (93 passing, was 59)

• tests/test_litellm/test_github_triage_with_llm.py — added a TestCloseCommentText class (pins the new "open a new PR" + @agent-shin reconsider wording) and 6 new TestTriageOrchestration cases for reconsider mode (skip-not-closed on open, reopen on pass, still-failing comment on fail, linked-issue short-circuit reopen, skip internal author in reconsider, reopen issue on pass).
• tests/test_litellm/test_github_close_low_quality_prs.py — replaced the "skip drafts" test with "closes drafts when score is low" + "closes brand-new PR when min_age=0" + "no skip when min_age=0". The "skip too young" assertion is preserved as opt-in.
• tests/test_litellm/test_github_triage_workflows.py — added triage_reconsider.yml to the destructive-gate guardrail table.

Total: 93 tests passing under uv run pytest tests/test_litellm/test_github_close_low_quality_prs.py tests/test_litellm/test_github_triage_with_llm.py tests/test_litellm/test_github_triage_workflows.py.

Pre-Submission checklist

• I have added testing in tests/test_litellm/ — test_github_triage_with_llm.py + test_github_close_low_quality_prs.py + test_github_triage_workflows.py.
• My PR passes the relevant unit tests (93 passed).
• Scope: triage-only; no edits to runtime LiteLLM code.

Type

🆕 New Feature
🚄 Infrastructure

Enablement plan (post-merge)

1. Add OPENAI_API_KEY to Settings → Secrets and variables → Actions → Secrets.
2. Optionally add repo variables TRIAGE_MODEL (default gpt-5.4-mini) and OPENAI_BASE_URL (if routing via the LiteLLM proxy).
3. Open a PR → the workflow runs in dry-run automatically. Check the workflow step summary to see the judge's verdict and proposed close-comment body. Iterate on the prompt (edit build_pr_prompt / build_issue_prompt) until the team is satisfied.
4. When ready, set repo variable AGENT_SHIN_ENABLED=true.
5. To smoke-test closing on a specific PR: gh workflow run "Agent Shin — PR triage" -f pr_number=NNN -f close=true.
6. After flipping on, the daily Greptile sweep will close any external-author PR with a Greptile score <4/5 (including drafts, regardless of age). Recommend a one-off gh workflow run "Close Low-Quality PRs" -f close=false first to preview the closure list in the step summary.

Out of scope (intentional)

• Doesn't close anything by default — every workflow is gated on AGENT_SHIN_ENABLED=true. Even with that set, automatic triggers (pull_request_target, issues) stay in dry-run; only manual workflow_dispatch with close=true, the daily scheduled sweep, or the explicit @agent-shin reconsider comment trigger are destructive.
• No retraining loop — verdicts are not logged anywhere persistent. If desired we can extend write_step_summary to also POST to a metrics endpoint.
• Greptile is not asked to re-review on closed PRs. The team asked whether Greptile re-reviews closed PRs; Greptile's docs don't say, so the close-comment wording does NOT promise behavior we can't verify. The contributor's path forward is either "open a new PR" (which Greptile definitely reviews) or @agent-shin reconsider (which re-runs the LiteLLM-side rubric judge, not Greptile).

Changes

• .github/pull_request_template.md — explicit rubric + dedicated fields + "open a new PR" / @agent-shin reconsider reopen guidance.
• .github/ISSUE_TEMPLATE/{bug_report,feature_request}.yml — required fields + auto-triage banner.
• .github/scripts/triage_with_llm.py — Agent Shin orchestrator + CLI; --reconsider mode with reopen_pr / reopen_issue and format_reopen_comment / format_reconsider_still_failing_comment.
• .github/scripts/close_low_quality_prs.py — Greptile-score auto-closer; no draft skip, default age filter 0, comment recommends "open a new PR" + @agent-shin reconsider.
• .github/workflows/triage_pr_with_llm.yml, .github/workflows/triage_issue_with_llm.yml — Agent Shin workflows.
• .github/workflows/close_low_quality_prs.yml — schedule for the Greptile-score sweep.
• .github/workflows/triage_reconsider.yml — NEW issue_comment-triggered reconsider workflow.
• tests/test_litellm/test_github_triage_with_llm.py — orchestration + comment-text tests.
• tests/test_litellm/test_github_close_low_quality_prs.py — evaluation logic tests.
• tests/test_litellm/test_github_triage_workflows.py — workflow YAML guardrails.

Slack Thread

  

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ mateo-berri
❌ cursoragent
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR ships Agent Shin, an LLM-as-judge triage system for external OSS contributions: a new triage_with_llm.py orchestrator (with reconsider mode), an updated close_low_quality_prs.py Greptile-score sweeper, four GitHub Actions workflows, updated contribution templates, and 93 unit tests.

• Agent Shin core (triage_with_llm.py): single-entrypoint LLM judge with full reconsider mode (@agent-shin reconsider comment trigger), cycle-anchored provenance check (was_auto_closed_by_agent_shin), fail-safe gating on every destructive path, and a dry-run mode honored in both regular and reconsider flows.
• Greptile sweeper (close_low_quality_prs.py): draft/age filter removed (default --min-age-days 0), 1000-PR cap emits a ::warning::, and fail-safe [] return on API errors preserves per-PR sweep continuity.
• Workflows: all four are gated on AGENT_SHIN_ENABLED == 'true'; OPENAI_API_KEY is withheld on public triggers until the variable is set; scheduled events are always dry-run; triage_reconsider.yml gates the LLM key on AGENT_SHIN_ENABLED (prior concern resolved in this version).

Confidence Score: 5/5

Safe to merge — all changes are confined to GitHub Actions triage infrastructure and contribution templates with no touch to runtime LiteLLM code.

Every destructive path is guarded by AGENT_SHIN_ENABLED, the --close flag, and a cycle-anchored provenance check. All concerns from previous review rounds are addressed in this version.

No files require special attention.

Important Files Changed

Filename	Overview
.github/scripts/triage_with_llm.py	New Agent Shin orchestrator: LLM-as-judge triage with reconsider mode, provenance checks, dry-run path, and fail-safe gating — all previously flagged concerns resolved in this version.
.github/scripts/close_low_quality_prs.py	New Greptile-score auto-closer: draft/age filter removed, fail-safe association check, cap-warning on 1000-PR limit.
.github/workflows/triage_reconsider.yml	New issue_comment-triggered reconsider workflow: OPENAI_API_KEY now gated on AGENT_SHIN_ENABLED == 'true' (prior concern addressed).
.github/workflows/close_low_quality_prs.yml	Updated daily Greptile-score sweep: scheduled events are always dry-run per intentional safety invariant.
tests/test_litellm/test_github_triage_with_llm.py	Extended unit tests: adds reconsider-mode cases, comment text pins, and linked-issue short-circuit reopen coverage.

_{Reviews (15): Last reviewed commit: "refactor(triage): share INTERNAL_ASSOCIA..." | Re-trigger Greptile}

[veria-ai]

PR overview

GitHub triage automation added

This PR adds Agent Shin and Greptile-based GitHub issue/PR triage workflows plus supporting scripts and tests. I reviewed the new write-capable workflows, pull_request_target usage, secret exposure gates, commenter authorization, bot-close provenance checks, and subprocess invocation paths; I did not find an actionable security issue.

Security review

• No new security issues were flagged in the latest review.
• No review issues remain open on this pull request.

Risk: 2/10

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[cursor]

Cursor Bugbot has reviewed your changes using high mode and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Close-comment formatters hardcode marker instead of using constant
  • Replaced the hardcoded "I'm **Agent Shin**" literal in format_pr_close_comment and format_issue_close_comment with f-string interpolation of AGENT_SHIN_AUTO_CLOSE_MARKER, matching the pattern used in close_low_quality_prs.py.

Preview (b16ebb5876)

diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml
--- a/.github/ISSUE_TEMPLATE/bug_report.yml
+++ b/.github/ISSUE_TEMPLATE/bug_report.yml
@@ -6,8 +6,11 @@
   - type: markdown
     attributes:
       value: |
-        Thanks for taking the time to fill out this bug report!
-        
+        Thanks for taking the time to file a bug report!
+
+        > ⚠️ **Auto-triage notice for external contributors:**
+        > Bug reports without **clear reproduction steps, expected vs. actual behavior, and a screenshot or terminal/log output** are auto-closed by our LLM triage bot with an explanation of what was missing. You can fill in the missing details and reopen at any time — the bot will re-evaluate. Internal BerriAI contributors are exempt.
+
         **💡 Tip:** See our [Troubleshooting Guide](https://docs.litellm.ai/docs/troubleshoot) for what information to include.
   - type: checkboxes
     id: duplicate-check
@@ -20,21 +23,33 @@
   - type: textarea
     id: what-happened
     attributes:
-      label: What happened?
-      description: Also tell us, what did you expect to happen?
-      placeholder: Tell us what you see!
-      value: "A bug happened!"
+      label: What happened? (Actual behavior)
+      description: A clear description of what is happening today, with the bug.
+      placeholder: e.g. "Calling completion() with model=gpt-4o-mini returns an empty string."
     validations:
       required: true
   - type: textarea
+    id: expected-behavior
+    attributes:
+      label: What did you expect to happen? (Expected behavior)
+      description: A clear description of what you expected to happen. **Required.**
+      placeholder: e.g. "I expected completion() to return the model's response text."
+    validations:
+      required: true
+  - type: textarea
     id: steps-to-reproduce
     attributes:
-      label: Steps to Reproduce
-      description: Please provide detailed steps to reproduce this bug(A curl/python code to reproduce the bug)
+      label: Steps to reproduce
+      description: |
+        Provide a minimal reproduction. Include a runnable Python snippet or a
+        `curl` command, your config.yaml if relevant, and the exact LiteLLM
+        version + Python version. Reports without a runnable reproduction are
+        auto-closed.
       placeholder: |
-        1. config.yaml file/ .env file/ etc.
-        2. Run the following code...
-        3. Observe the error...
+        1. Create `config.yaml` with: ...
+        2. Start the proxy with: `litellm --config config.yaml --port 4000`
+        3. Run this Python / curl: ...
+        4. Observe: ...
       value: |
         1. 
         2. 
@@ -44,9 +59,15 @@
   - type: textarea
     id: logs
     attributes:
-      label: Relevant log output
-      description: Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks.
+      label: Relevant log output / screenshot
+      description: |
+        **Required.** Paste the full traceback, stderr, proxy logs, or attach a
+        screenshot showing the bug. For UI bugs a screenshot or screen
+        recording is mandatory. Without proof of the bug, the issue is
+        auto-closed.
       render: shell
+    validations:
+      required: true
   - type: dropdown
     id: component
     attributes:
@@ -63,14 +84,14 @@
   - type: input
     id: version
     attributes:
-      label: What LiteLLM version are you on ? 
+      label: What LiteLLM version are you on ?
       placeholder: v1.53.1
     validations:
       required: true
   - type: input
     id: contact
     attributes:
-      label: Twitter / LinkedIn details 
+      label: Twitter / LinkedIn details
       description: We announce new features on Twitter + LinkedIn. If this issue leads to an announcement, and you'd like a mention, we'll gladly shout you out!
       placeholder: ex. @krrish_dh / https://www.linkedin.com/in/krish-d/
     validations:

diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml
--- a/.github/ISSUE_TEMPLATE/feature_request.yml
+++ b/.github/ISSUE_TEMPLATE/feature_request.yml
@@ -1,4 +1,4 @@
-name: 🚀 Feature Request 
+name: 🚀 Feature Request
 description: Submit a proposal/request for a new LiteLLM feature.
 title: "[Feature]: "
 labels: ["enhancement"]
@@ -6,7 +6,10 @@
   - type: markdown
     attributes:
       value: |
-        Thanks for making LiteLLM better! 
+        Thanks for making LiteLLM better!
+
+        > ⚠️ **Auto-triage notice for external contributors:**
+        > Feature requests need (1) a clear description of the proposed feature, (2) the motivation / use case with a concrete example, and (3) what success looks like. Vague requests are auto-closed by our LLM triage bot with an explanation. Fill in the missing details and reopen at any time — the bot will re-evaluate. Internal BerriAI contributors are exempt.
   - type: checkboxes
     id: duplicate-check
     attributes:
@@ -18,16 +21,25 @@
   - type: textarea
     id: the-feature
     attributes:
-      label: The Feature
-      description: A clear and concise description of the feature proposal
-      placeholder: Tell us what you want!
+      label: The feature
+      description: A clear and concise description of the feature proposal. What should LiteLLM do that it doesn't today?
+      placeholder: e.g. "Support per-team max_input_tokens overrides on the proxy."
     validations:
       required: true
   - type: textarea
     id: motivation
     attributes:
-      label: Motivation, pitch
-      description: Please outline the motivation for the proposal. Is your feature request related to a specific problem? e.g., "I'm working on X and would like Y to be possible". If this is related to another GitHub issue, please link here too.
+      label: Motivation, pitch, and concrete example
+      description: |
+        **Required.** Why is this needed? Include a concrete use case — what
+        you're trying to accomplish, what's blocked today, and what success
+        would look like (ideally with an example config / API call / UI flow).
+        If this is related to another GitHub issue, link it here too.
+      placeholder: |
+        I'm running a multi-tenant proxy where team A processes long docs and
+        team B only does short chats. Today I have to spin up two proxies.
+        With this feature I could set max_input_tokens per team and route in one
+        proxy. Example config: ...
     validations:
       required: true
   - type: dropdown
@@ -56,7 +68,7 @@
   - type: input
     id: contact
     attributes:
-      label: Twitter / LinkedIn details 
+      label: Twitter / LinkedIn details
       description: We announce new features on Twitter + LinkedIn. When this is announced, and you'd like a mention, we'll gladly shout you out!
       placeholder: ex. @krrish_dh / https://www.linkedin.com/in/krish-d/
     validations:

diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
--- a/.github/pull_request_template.md
+++ b/.github/pull_request_template.md
@@ -1,11 +1,67 @@
+<!--
+👋 Hi there — please read before submitting.
+
+To keep the review queue healthy for everyone, **every external PR is
+auto-triaged** by an LLM bot ("Agent Shin") on open / reopen, regardless of
+whether the PR is a draft or marked ready for review. PRs that don't meet
+the rubric below are auto-closed with an explanation.
+
+To pass triage, your PR must satisfy AT LEAST ONE of:
+
+  (A) Link a related GitHub issue (e.g. "Fixes #1234" or "Resolves
+      https://github.com/BerriAI/litellm/issues/1234"), OR
+
+  (B) Provide ALL of the following IN THIS PR DESCRIPTION:
+      - A clear problem description (what bug or missing feature this addresses)
+      - Expected vs. actual behavior
+      - Visual QA proof (before/after screenshots, screen recording, or
+        terminal output demonstrating that the fix/feature works end-to-end)
+
+Every external PR (including drafts, regardless of age) also receives a
+Greptile code review. Any PR with a Greptile Confidence Score below 4/5 is
+auto-closed.
+
+If your PR was auto-closed and you've addressed the feedback, you have two
+options to bring it back:
+
+  - **Open a new PR** with the updated branch (recommended — GitHub does not
+    let external contributors reopen a PR that was closed by a bot or
+    maintainer).
+  - **Or** comment `@agent-shin reconsider` on the closed PR. Agent Shin
+    will re-run triage and reopen the PR if it now meets the bar.
+
+Internal BerriAI contributors are exempt from this auto-triage — fill in the
+Linear ticket section instead.
+-->
+
 ## Relevant issues
 
-<!-- e.g. "Fixes #000" -->
+<!-- e.g. "Fixes #000". If you have no related issue, fill in the
+"Problem description / Expected vs. Actual / QA proof" sections below. -->
 
 ## Linear ticket
 
-<!-- if you are an internal contributor, add the Linear ticket e.g. "Resolves LIT-1234" to magically link the Linear ticket to the GitHub PR -->
+<!-- INTERNAL CONTRIBUTORS ONLY: add the Linear ticket e.g. "Resolves LIT-1234"
+to magically link the Linear ticket to the GitHub PR. External contributors:
+leave this blank and fill in the problem/expected-actual/QA sections below. -->
 
+## Problem description
+
+<!-- What bug or missing feature does this PR address? One or two paragraphs.
+External contributors: required unless you linked a GitHub issue above. -->
+
+## Expected vs. actual behavior
+
+<!-- What did you expect to happen? What is happening today (before this PR)?
+External contributors: required unless you linked a GitHub issue above. -->
+
+## QA proof
+
+<!-- Required for external contributors: include before/after screenshots,
+a screen recording, or terminal/log output that demonstrates the fix or feature
+works end-to-end. For UI changes, before/after screenshots are mandatory.
+For backend changes, terminal output of a passing test or curl command is fine. -->
+
 ## Pre-Submission checklist
 
 **Please complete all items before asking a LiteLLM maintainer to review your PR**
@@ -13,7 +69,7 @@
 - [ ] I have Added testing in the [`tests/test_litellm/`](https://github.com/BerriAI/litellm/tree/main/tests/test_litellm) directory, **Adding at least 1 test is a hard requirement** - [see details](https://docs.litellm.ai/docs/extras/contributing_code)
 - [ ] My PR passes all unit tests on [`make test-unit`](https://docs.litellm.ai/docs/extras/contributing_code)
 - [ ] My PR's scope is as isolated as possible, it only solves 1 specific problem
-- [ ] I have requested a Greptile review by commenting `@greptileai` and received a **Confidence Score of at least 4/5** before requesting a maintainer review
+- [ ] I have received a Greptile **Confidence Score of at least 4/5** before requesting a maintainer review (Greptile reviews automatically on open; comment `@greptileai` to re-trigger after pushing fixes)
 
 ## Delays in PR merge?
 
@@ -36,13 +92,6 @@
 - [ ] **Merge / cherry-pick CI run**  
        Links:
 
-## Screenshots / Proof of Fix
-
-<!-- Include screenshots, screen recordings, or log output demonstrating that your changes work as expected.
-     For bug fixes: show reproduction before the fix and passing behavior after.
-     For new features: show the feature working end-to-end.
-     For UI changes: include before/after screenshots. -->
-
 ## Type
 
 <!-- Select the type of Pull Request -->

diff --git a/.github/scripts/close_low_quality_prs.py b/.github/scripts/close_low_quality_prs.py
new file mode 100644
--- /dev/null
+++ b/.github/scripts/close_low_quality_prs.py
@@ -1,0 +1,467 @@
+#!/usr/bin/env python3
+"""
+Auto-close low-quality pull requests.
+
+Closes open PRs (including drafts, regardless of age) that satisfy ALL of:
+  1. Have a Greptile (`greptile-apps`) review comment whose latest
+     "Confidence Score: X/5" is below the configured threshold (default: 4).
+  2. Are authored by an external OSS contributor (internal BerriAI
+     contributors are exempt).
+  3. Do not carry an opt-out label (default: "do not close").
+
+`--min-age-days` is retained as an opt-in safety net for one-off backfill
+runs (default: 0). The team's intent is that the count of open PRs equals
+the count of PRs internal collaborators need to action on, so neither age
+nor draft status acts as a free pass.
+
+For each match, the script posts an explanatory comment and closes the PR.
+Because OSS contributors *cannot* reopen a PR closed by the bot/maintainer
+(GitHub limitation), the close-comment instructs them to push their fixes
+and **open a fresh PR**, or to comment `@agent-shin reconsider` on the
+closed PR to have the LLM judge re-evaluate (and reopen on pass).
+
+Requires the `gh` CLI to be authenticated.
+
+Usage examples:
+    # Dry run (default) - prints what would be closed
+    python3 close_low_quality_prs.py
+
+    # Actually close matching PRs
+    python3 close_low_quality_prs.py --close
+
+    # Restrict to PRs at least N days old (one-off backfill safety net)
+    python3 close_low_quality_prs.py --min-age-days 7 --min-score 4 --close
+"""
+
+from __future__ import annotations
+
+import argparse
+import datetime as dt
+import json
+import re
+import subprocess
+import sys
+from pathlib import Path
+from typing import Iterable
+
+# Share the auto-close marker with the sibling Agent Shin script instead of
+# duplicating the literal — the reconsider provenance check
+# (`was_auto_closed_by_agent_shin`) keys off this exact phrase, so a drift
+# between the two files would silently break reconsider for Greptile-closed
+# PRs without any test catching it.
+_SCRIPTS_DIR = Path(__file__).resolve().parent
+if str(_SCRIPTS_DIR) not in sys.path:
+    sys.path.insert(0, str(_SCRIPTS_DIR))
+from triage_with_llm import AGENT_SHIN_AUTO_CLOSE_MARKER  # noqa: E402
+
+# Greptile's GitHub App appears as `greptile-apps[bot]` in REST API comments
+# and `greptile-apps` in `gh pr view --json` output. Accept either form.
+GREPTILE_BOT_LOGINS = frozenset({"greptile-apps", "greptile-apps[bot]"})
+
+# Matches lines like:
+#   <h3>Confidence Score: 3/5</h3>
+#   **Confidence Score: 4/5**
+#   Confidence Score: 5 / 5
+SCORE_PATTERN = re.compile(
+    r"confidence\s*score\s*[:\-]?\s*(\d+)\s*/\s*5",
+    re.IGNORECASE,
+)
+
+# `author_association` values for internal BerriAI contributors who should be
+# exempt from auto-triage.
+INTERNAL_AUTHOR_ASSOCIATIONS = frozenset({"OWNER", "MEMBER", "COLLABORATOR"})
+
+# Default labels that exempt a PR from auto-close. Defined at module scope (not
+# as a mutable argparse default) so that `--optout-label foo` REPLACES the
+# defaults instead of appending to them — the argparse `action="append"` +
+# `default=[...]` combination silently mutates the shared default list.
+DEFAULT_OPTOUT_LABELS = ("do not close", "keep open", "wip")
+
+
+def gh(*args: str) -> str:
+    """Run a `gh` CLI command and return stdout. Raises on non-zero exit."""
+    result = subprocess.run(
+        ["gh", *args],
+        capture_output=True,
+        text=True,
+        check=True,
+    )
+    return result.stdout
+
+
+# `gh pr list --limit` caps at 1000 (the CLI's documented hard ceiling).
+# Surface a warning if we ever hit that cap so the silent truncation is
+# visible in workflow logs instead of just being a missed close.
+GH_PR_LIST_LIMIT = 1000
+
+
+def fetch_open_prs(repo: str | None) -> list[dict]:
+    """Fetch all open PRs (number, createdAt, isDraft, labels, author).
+
+    Includes drafts: `gh pr list --state open` returns both ready-for-review
+    and draft PRs by default. This is the desired behavior — drafts are not
+    a free pass; the internal-collaborator open-PR queue should reflect every
+    PR that needs human attention regardless of draft status.
+    """
+    repo_args = ["--repo", repo] if repo else []
+    fields = "number,title,createdAt,isDraft,labels,author,url"
+    raw = gh(
+        "pr",
+        "list",
+        "--state",
+        "open",
+        "--limit",
+        str(GH_PR_LIST_LIMIT),
+        "--json",
+        fields,
+        *repo_args,
+    )
+    prs = json.loads(raw)
+    if len(prs) >= GH_PR_LIST_LIMIT:
+        # `gh pr list --limit N` returns at most N rows even if more exist;
+        # log a GitHub Actions warning so the truncation isn't silent.
+        message = (
+            f"fetch_open_prs hit the gh CLI cap ({GH_PR_LIST_LIMIT}); "
+            "the open-PR list is likely truncated. Switch to paginated "
+            "`gh api` calls if the repo regularly exceeds this cap."
+        )
+        print(f"::warning::{message}", file=sys.stderr)
+    return prs
+
+
+def fetch_pr_author_association(pr_number: int, repo: str | None) -> str:
+    """Return the GitHub `author_association` for a PR, uppercase.
+
+    Values: OWNER, MEMBER, COLLABORATOR, CONTRIBUTOR, FIRST_TIME_CONTRIBUTOR,
+    FIRST_TIMER, MANNEQUIN, NONE. Returns "" on lookup failure.
+    """
+    endpoint = (
+        f"repos/{repo}/pulls/{pr_number}"
+        if repo
+        else f"repos/{{owner}}/{{repo}}/pulls/{pr_number}"
+    )
+    try:
+        data = json.loads(gh("api", endpoint))
+    except subprocess.CalledProcessError:
+        return ""
+    return (data.get("author_association") or "").upper()
+
+
+def is_external_pr_author(pr: dict, repo: str | None) -> bool:
+    """Return True if the PR author is an external OSS contributor.
+
+    Internal = `OWNER` / `MEMBER` / `COLLABORATOR` association, or a bot login.
+    """
+    login = ((pr.get("author") or {}).get("login") or "").lower()
+    if login.endswith("[bot]") or login in {"dependabot", "github-actions"}:
+        return False
+    association = fetch_pr_author_association(pr["number"], repo)
+    # Fail-safe: if the API lookup failed (empty string), treat the author as
+    # internal so we don't auto-close their PR. Auto-close is destructive, so
+    # an unknown association should never make a PR eligible for closing.
+    if not association or association in INTERNAL_AUTHOR_ASSOCIATIONS:
+        return False
+    return True
+
+
+def fetch_pr_comments(pr_number: int, repo: str | None) -> list[dict]:
+    """Fetch issue-level comments on a PR (where Greptile posts its summary).
+
+    Returns [] on API failure so a transient hiccup on any single PR doesn't
+    abort the whole daily sweep mid-loop. Matches the fail-safe pattern in
+    `fetch_pr_author_association`; downstream the empty list becomes a
+    `skip-no-greptile-score` action and the PR is re-evaluated on the next run.
+    """
+    endpoint = (
+        f"repos/{repo}/issues/{pr_number}/comments?per_page=100"
+        if repo
+        else f"repos/{{owner}}/{{repo}}/issues/{pr_number}/comments?per_page=100"
+    )
+    try:
+        raw = gh("api", "--paginate", endpoint)
+    except subprocess.CalledProcessError:
+        return []
+    comments: list[dict] = []
+    for line in raw.strip().splitlines():
+        line = line.strip()
+        if not line:
+            continue
+        try:
+            parsed = json.loads(line)
+        except json.JSONDecodeError:
+            return []
+        if isinstance(parsed, list):
+            comments.extend(parsed)
+        else:
+            comments.append(parsed)
+    return comments
+
+
+def extract_greptile_score(comments: Iterable[dict]) -> tuple[int, dict] | None:
+    """Return (score, comment) for the most recent Greptile-authored comment
+    that contains a "Confidence Score: X/5". Returns None if no such comment.
+
+    "Most recent" is determined by the comment's `updated_at` (falling back to
+    `created_at`), so re-reviews override earlier passes.
+    """
+    candidates: list[tuple[str, int, dict]] = []
+    for comment in comments:
+        user = (comment.get("user") or {}).get("login", "")
+        if user not in GREPTILE_BOT_LOGINS:
+            continue
+        body = comment.get("body") or ""
+        match = SCORE_PATTERN.search(body)
+        if not match:
+            continue
+        score = int(match.group(1))
+        timestamp = comment.get("updated_at") or comment.get("created_at") or ""
+        candidates.append((timestamp, score, comment))
+
+    if not candidates:
+        return None
+
+    candidates.sort(key=lambda triple: triple[0])
+    _, score, comment = candidates[-1]
+    return score, comment
+
+
+def parse_iso8601(value: str) -> dt.datetime:
+    """Parse a GitHub ISO-8601 timestamp into a timezone-aware datetime."""
+    return dt.datetime.fromisoformat(value.replace("Z", "+00:00"))
+
+
+def has_optout_label(pr: dict, optout_labels: set[str]) -> bool:
+    labels = {label.get("name", "").lower() for label in pr.get("labels", [])}
+    return bool(labels & {lbl.lower() for lbl in optout_labels})
+
+
+def close_pr(
+    pr: dict,
+    score: int,
+    threshold: int,
+    age_days: int,
+    repo: str | None,
+    dry_run: bool,
+    label: str | None,
+) -> None:
+    """Post the explanatory comment and close the PR."""
+    pr_number = pr["number"]
+    repo_args = ["--repo", repo] if repo else []
+
+    if dry_run:
+        print(
+            f"  [DRY RUN] Would close PR #{pr_number} "
+            f"(age={age_days}d, greptile={score}/5): {pr['title']}"
+        )
+        return
+
+    comment_body = (
+        f"👋 Hi, thanks for the PR! {AGENT_SHIN_AUTO_CLOSE_MARKER}, the automated triage "
+        "bot for this repository. Closing as part of automated PR triage.\n\n"
+        f"Greptile's most recent review scored this PR **{score}/5**, below "
+        f"our merge bar of **{threshold}/5**.\n\n"
+        "We close low-confidence PRs aggressively to keep the review queue "
+        "manageable for maintainers and contributors alike. **This is not a "
+        "rejection of the idea** — to bring this back:\n\n"
+        "1. Push the fixes that address Greptile's feedback (continue using "
+        "your existing branch is fine).\n"
+        "2. **Open a new PR** with the updated branch. Greptile will review "
+        "it again, and if it scores "
+        f"**{threshold}/5 or higher** a maintainer will take another look.\n\n"
+        "_Why open a new PR instead of reopening this one?_ GitHub does not "
+        "let external contributors reopen a PR that was closed by a bot or "
+        "maintainer, so a fresh PR is the most reliable path forward. If you "
+        "would prefer this exact PR re-evaluated, comment "
+        "`@agent-shin reconsider` once you've pushed the fixes — Agent Shin "
+        "will re-run triage and reopen this PR if it now meets the bar.\n\n"
+        "Thanks for contributing to LiteLLM. We know auto-closures can sting; "
+        "the goal is to keep the project healthy, not to dismiss your work."
+    )
+    gh("pr", "comment", str(pr_number), "--body", comment_body, *repo_args)
+
+    if label:
+        try:
+            gh("pr", "edit", str(pr_number), "--add-label", label, *repo_args)
+        except subprocess.CalledProcessError as exc:
+            stderr = (exc.stderr or "").strip()
+            print(f"  warn: failed to add label '{label}' to #{pr_number}: {stderr}")
+
+    gh("pr", "close", str(pr_number), *repo_args)
+    print(f"  Closed PR #{pr_number} (greptile={score}/5, age={age_days}d)")
+
+
+def evaluate_pr(
+    pr: dict,
+    now: dt.datetime,
+    min_age_days: int,
+    min_score: int,
+    repo: str | None,
+    optout_labels: set[str],
+) -> tuple[str, int | None, int | None]:
+    """Decide whether to close `pr`.
+
+    Returns (action, score_or_none, age_days_or_none) where action is one of:
+        "skip-too-young", "skip-optout-label", "skip-internal",
+        "skip-no-greptile-score", "skip-score-ok", or "close".
+
+    Drafts are NOT skipped — the goal is "open PR count == PRs internal
+    collaborators need to action on", and a draft that Greptile scored <4/5
+    is still in that queue. Authors can opt out via the `wip` label (see
+    `DEFAULT_OPTOUT_LABELS`) if they need to keep a long-lived draft open.
+    """
+    if has_optout_label(pr, optout_labels):
+        return ("skip-optout-label", None, None)
+
+    created = parse_iso8601(pr["createdAt"])
+    age_days = (now - created).days
+    # `min_age_days` defaults to 0 (close as soon as Greptile scores low).
+    # Set a positive value via --min-age-days for one-off backfill runs that
+    # want to skip very-young PRs.
+    if min_age_days > 0 and age_days < min_age_days:
+        return ("skip-too-young", None, age_days)
+
+    # Only auto-close external OSS contributors. Internal contributors
+    # (BerriAI org members) handle their own backlog.
+    if not is_external_pr_author(pr, repo):
+        return ("skip-internal", None, age_days)
+
+    comments = fetch_pr_comments(pr["number"], repo)
+    extraction = extract_greptile_score(comments)
+    if extraction is None:
+        return ("skip-no-greptile-score", None, age_days)
+
+    score, _ = extraction
+    if score >= min_score:
+        return ("skip-score-ok", score, age_days)
+
+    return ("close", score, age_days)
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "--repo",
+        type=str,
+        default=None,
+        help="Repository (owner/repo). Auto-detected if omitted.",
+    )
+    parser.add_argument(
+        "--min-age-days",
+        type=int,
+        default=0,
+        help=(
+            "Minimum age (in days) before a PR is eligible. Default 0 = "
+            "close as soon as Greptile flags it. Set a positive value for "
+            "one-off backfill runs that want to spare very-young PRs."
+        ),
+    )
+    parser.add_argument(
+        "--min-score",
+        type=int,
+        default=4,
+        choices=range(1, 6),
+        help="Greptile score below which a PR is closed (default: 4 -> closes <4/5).",
+    )
+    parser.add_argument(
+        "--optout-label",
+        action="append",
+        default=None,
+        help=(
+            "Label(s) that exempt a PR from auto-close. Repeat to add more. "
+            "Case-insensitive. When omitted, defaults to "
+            f"{list(DEFAULT_OPTOUT_LABELS)!r}; passing this flag REPLACES the "
+            "defaults (argparse `append` with a mutable default would append "
+            "instead, which we explicitly avoid)."
+        ),
+    )
+    parser.add_argument(
+        "--close-label",
+        type=str,
+        default=None,
+        help=(
+            "Optional label to add to PRs that get auto-closed "
+            "(e.g. 'auto-closed-low-quality'). Must already exist on the repo."
+        ),
+    )
+    parser.add_argument(
+        "--close",
+        action="store_true",
+        help="Actually close matching PRs (default is dry-run).",
+    )
+    parser.add_argument(
+        "--limit",
+        type=int,
+        default=None,
+        help=(
+            "Maximum number of PRs to close in one run (safety net). "
+            "Applied in dry-run too so `--limit N` previews exactly the "
+            "first N closures."
+        ),
+    )
+    args = parser.parse_args()
+
+    dry_run = not args.close
+    if dry_run:
+        print("=== DRY RUN MODE (pass --close to actually close PRs) ===\n")
+
+    print("Fetching open PRs...")
+    prs = fetch_open_prs(args.repo)
+    print(f"Found {len(prs)} open PRs.\n")
+
+    now = dt.datetime.now(dt.timezone.utc)
+    optout_labels = set(args.optout_label or DEFAULT_OPTOUT_LABELS)
+
+    closed = 0
+    summary = {
+        "close": 0,
+        "skip-too-young": 0,
+        "skip-optout-label": 0,
+        "skip-internal": 0,
+        "skip-no-greptile-score": 0,
+        "skip-score-ok": 0,
+    }
+
+    for pr in sorted(prs, key=lambda p: p["createdAt"]):
+        action, score, age_days = evaluate_pr(
+            pr,
+            now,
+            args.min_age_days,
+            args.min_score,
+            args.repo,
+            optout_labels,
+        )
+        summary[action] = summary.get(action, 0) + 1
+
+        if action != "close":
+            continue
+
+        assert score is not None and age_days is not None
+        print(
+            f"#{pr['number']}: \"{pr['title']}\" "
+            f"(age={age_days}d, greptile={score}/5) -> close"
+        )
+        close_pr(
+            pr,
+            score=score,
+            threshold=args.min_score,
+            age_days=age_days,
+            repo=args.repo,
+            dry_run=dry_run,
+            label=args.close_label,
+        )
+
+        closed += 1
+        if args.limit is not None and closed >= args.limit:
+            verb = "Would close" if dry_run else "Closed"
+            print(f"\nReached --limit={args.limit} ({verb} count); stopping.")
+            break
+
+    print("\n=== Summary ===")
+    for key, value in summary.items():
+        print(f"  {key:28s} {value}")
+    print(f"\nTotal {'would close' if dry_run else 'closed'}: {summary['close']}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

diff --git a/.github/scripts/triage_with_llm.py b/.github/scripts/triage_with_llm.py
new file mode 100644
--- /dev/null
+++ b/.github/scripts/triage_with_llm.py
@@ -1,0 +1,908 @@
+#!/usr/bin/env python3
+"""
+Agent Shin — LLM-as-judge triage for external OSS pull requests and issues.
+
+Evaluates a single PR or issue against the contribution rubric and, when the
+LLM judge marks it as failing, posts an explanatory comment + closes the
+PR/issue. Re-triggers on `reopened` so contributors can iterate back in by
+filling in the missing pieces and reopening.
+
+Internal BerriAI contributors (`author_association` in {OWNER, MEMBER,
+COLLABORATOR}) and bot accounts are skipped entirely.
+
+Usage:
+    triage_with_llm.py --repo owner/repo --pr 1234
+    triage_with_llm.py --repo owner/repo --issue 5678
+    triage_with_llm.py --repo owner/repo --pr 1234 --close    # actually close
+    triage_with_llm.py --repo owner/repo --pr 1234 --print-prompt  # show prompt
+
+Defaults are SAFE: without `--close` the script writes a verdict to stdout (and,
+when running in GitHub Actions, to $GITHUB_STEP_SUMMARY) but takes no GitHub
+write actions.
+
+Environment:
+    GH_TOKEN / GITHUB_TOKEN  - for `gh` CLI auth (auto-set in Actions)
+    OPENAI_API_KEY           - required when --close is passed
+    OPENAI_BASE_URL          - optional (route to any OpenAI-compatible API)
+    TRIAGE_MODEL             - optional model override (default: gpt-5.4-mini)
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import subprocess
+import sys
+import textwrap
+from typing import Any
+
+DEFAULT_MODEL = "gpt-5.4-mini"
+
+INTERNAL_ASSOCIATIONS = frozenset({"OWNER", "MEMBER", "COLLABORATOR"})
+
+# Marker phrase Agent Shin always includes in its auto-close comments
+# (see `format_pr_close_comment` / `format_issue_close_comment`). The
+# provenance check for reconsider matches this marker against a comment
+# authored by the same bot login that performed the most recent `closed`
+# event, so a contributor cannot reopen a PR/issue that a maintainer
+# closed after a prior Agent Shin auto-close. Keep the marker in sync
+# with the literal text in those formatter functions.
+AGENT_SHIN_AUTO_CLOSE_MARKER = "I'm **Agent Shin**"
+
+# Model families that require `reasoning_effort` to be set, and that reject
+# `temperature != 1` unless `reasoning_effort` is "none". For these models we
+# pass `reasoning_effort="none"` so a `temperature=0` deterministic judgment
+# is still accepted. See litellm/llms/openai/chat/gpt_5_transformation.py for
+# the full set of constraints LiteLLM applies to these models.
+GPT5_FAMILY_PREFIX = "gpt-5"
+
+# Regexes for picking off "obvious passes" without burning LLM tokens.
+#
+# Keep this list to GitHub's documented PR-closing keywords only
+# (https://docs.github.com/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue).
+# Casual mentions like "see #1234" or "ref #1234" are intentionally NOT
+# auto-passed — they should fall through to the LLM judge, which has the
+# stricter rubric "a bare issue number without a closing keyword counts only
... diff truncated: showing 800 of 4062 lines

_{You can send follow-ups to the cloud agent here.}

[mateo-berri]

@greptileai

[greptile-apps]

fetch_pr_comments returns [] early on any JSONDecodeError, discarding all comments collected from earlier paginated pages. If gh api --paginate delivers the Greptile score on page 1 and a malformed line appears on page 2, the score is silently lost and the PR gets skip-no-greptile-score instead of being evaluated correctly. The sibling fetch_issue_comments in triage_with_llm.py avoids this by using continue to skip the bad line and keep going.

Suggested change

	try:
	parsed = json.loads(line)
	except json.JSONDecodeError:
	return []
	try:
	parsed = json.loads(line)
	except json.JSONDecodeError:
	continue

[mateo-berri]

Same point as in #28117 (comment) — answering inline so the thread isn't lost:

The two functions have deliberately opposite fail-safe semantics, which is why they differ:

• fetch_issue_comments (triage_with_llm.py) feeds the reconsider provenance check. Its docstring even spells this out: "Returns [] on error (the reconsider path treats 'no comments found' as 'no proof Agent Shin closed this', which fails-safe)." If a malformed later page makes us think no auto-close marker exists, the reconsider path refuses to reopen — non-destructive. So preserving earlier pages with continue is correct.
• fetch_pr_comments (close_low_quality_prs.py) feeds a destructive sweep. extract_greptile_score picks the most recent Greptile-authored score by updated_at. If a later page is malformed, continue would let an older/stale higher score slip through and silently miss a re-review that just dropped the PR to <4/5 — i.e. close the wrong PR, or skip closing one we should close. return [] collapses that ambiguity into skip-no-greptile-score, the PR is left alone, and it's re-evaluated on tomorrow's run. That is the safer outcome for the destructive direction.

The "discards valid earlier pages" framing reads like a bug only if the function's job were to maximize comment yield. Its actual job here is to be a confidence gate on a destructive action; under partial pagination data the correct answer is "I don't know, skip", not "close based on whatever I happened to parse." Pinned by TestFetchPrComments::test_should_return_empty_list_when_paginated_json_is_malformed (added in c5d5968) so this contract doesn't drift.

Leaving as-is.

[mateo-berri]

@greptileai

[cursor]

Cursor Bugbot has reviewed your changes using high mode and found 1 potential issue.

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Resolved by another fix: Provenance check defeated by shared github-actions[bot] identity
  • A separate commit (dc80aa8) already on the branch anchors the marker comment to the current open→closed cycle (must post-date the latest reopen and not exceed the latest close) and adds a stale-workflow regression test, closing the shared-identity gap.

Preview (40da407e8e)

diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml
--- a/.github/ISSUE_TEMPLATE/bug_report.yml
+++ b/.github/ISSUE_TEMPLATE/bug_report.yml
@@ -6,8 +6,11 @@
   - type: markdown
     attributes:
       value: |
-        Thanks for taking the time to fill out this bug report!
-        
+        Thanks for taking the time to file a bug report!
+
+        > ⚠️ **Auto-triage notice for external contributors:**
+        > Bug reports without **clear reproduction steps, expected vs. actual behavior, and a screenshot or terminal/log output** are auto-closed by our LLM triage bot with an explanation of what was missing. You can fill in the missing details and reopen at any time — the bot will re-evaluate. Internal BerriAI contributors are exempt.
+
         **💡 Tip:** See our [Troubleshooting Guide](https://docs.litellm.ai/docs/troubleshoot) for what information to include.
   - type: checkboxes
     id: duplicate-check
@@ -20,21 +23,33 @@
   - type: textarea
     id: what-happened
     attributes:
-      label: What happened?
-      description: Also tell us, what did you expect to happen?
-      placeholder: Tell us what you see!
-      value: "A bug happened!"
+      label: What happened? (Actual behavior)
+      description: A clear description of what is happening today, with the bug.
+      placeholder: e.g. "Calling completion() with model=gpt-4o-mini returns an empty string."
     validations:
       required: true
   - type: textarea
+    id: expected-behavior
+    attributes:
+      label: What did you expect to happen? (Expected behavior)
+      description: A clear description of what you expected to happen. **Required.**
+      placeholder: e.g. "I expected completion() to return the model's response text."
+    validations:
+      required: true
+  - type: textarea
     id: steps-to-reproduce
     attributes:
-      label: Steps to Reproduce
-      description: Please provide detailed steps to reproduce this bug(A curl/python code to reproduce the bug)
+      label: Steps to reproduce
+      description: |
+        Provide a minimal reproduction. Include a runnable Python snippet or a
+        `curl` command, your config.yaml if relevant, and the exact LiteLLM
+        version + Python version. Reports without a runnable reproduction are
+        auto-closed.
       placeholder: |
-        1. config.yaml file/ .env file/ etc.
-        2. Run the following code...
-        3. Observe the error...
+        1. Create `config.yaml` with: ...
+        2. Start the proxy with: `litellm --config config.yaml --port 4000`
+        3. Run this Python / curl: ...
+        4. Observe: ...
       value: |
         1. 
         2. 
@@ -44,9 +59,15 @@
   - type: textarea
     id: logs
     attributes:
-      label: Relevant log output
-      description: Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks.
+      label: Relevant log output / screenshot
+      description: |
+        **Required.** Paste the full traceback, stderr, proxy logs, or attach a
+        screenshot showing the bug. For UI bugs a screenshot or screen
+        recording is mandatory. Without proof of the bug, the issue is
+        auto-closed.
       render: shell
+    validations:
+      required: true
   - type: dropdown
     id: component
     attributes:
@@ -63,14 +84,14 @@
   - type: input
     id: version
     attributes:
-      label: What LiteLLM version are you on ? 
+      label: What LiteLLM version are you on ?
       placeholder: v1.53.1
     validations:
       required: true
   - type: input
     id: contact
     attributes:
-      label: Twitter / LinkedIn details 
+      label: Twitter / LinkedIn details
       description: We announce new features on Twitter + LinkedIn. If this issue leads to an announcement, and you'd like a mention, we'll gladly shout you out!
       placeholder: ex. @krrish_dh / https://www.linkedin.com/in/krish-d/
     validations:

diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml
--- a/.github/ISSUE_TEMPLATE/feature_request.yml
+++ b/.github/ISSUE_TEMPLATE/feature_request.yml
@@ -1,4 +1,4 @@
-name: 🚀 Feature Request 
+name: 🚀 Feature Request
 description: Submit a proposal/request for a new LiteLLM feature.
 title: "[Feature]: "
 labels: ["enhancement"]
@@ -6,7 +6,10 @@
   - type: markdown
     attributes:
       value: |
-        Thanks for making LiteLLM better! 
+        Thanks for making LiteLLM better!
+
+        > ⚠️ **Auto-triage notice for external contributors:**
+        > Feature requests need (1) a clear description of the proposed feature, (2) the motivation / use case with a concrete example, and (3) what success looks like. Vague requests are auto-closed by our LLM triage bot with an explanation. Fill in the missing details and reopen at any time — the bot will re-evaluate. Internal BerriAI contributors are exempt.
   - type: checkboxes
     id: duplicate-check
     attributes:
@@ -18,16 +21,25 @@
   - type: textarea
     id: the-feature
     attributes:
-      label: The Feature
-      description: A clear and concise description of the feature proposal
-      placeholder: Tell us what you want!
+      label: The feature
+      description: A clear and concise description of the feature proposal. What should LiteLLM do that it doesn't today?
+      placeholder: e.g. "Support per-team max_input_tokens overrides on the proxy."
     validations:
       required: true
   - type: textarea
     id: motivation
     attributes:
-      label: Motivation, pitch
-      description: Please outline the motivation for the proposal. Is your feature request related to a specific problem? e.g., "I'm working on X and would like Y to be possible". If this is related to another GitHub issue, please link here too.
+      label: Motivation, pitch, and concrete example
+      description: |
+        **Required.** Why is this needed? Include a concrete use case — what
+        you're trying to accomplish, what's blocked today, and what success
+        would look like (ideally with an example config / API call / UI flow).
+        If this is related to another GitHub issue, link it here too.
+      placeholder: |
+        I'm running a multi-tenant proxy where team A processes long docs and
+        team B only does short chats. Today I have to spin up two proxies.
+        With this feature I could set max_input_tokens per team and route in one
+        proxy. Example config: ...
     validations:
       required: true
   - type: dropdown
@@ -56,7 +68,7 @@
   - type: input
     id: contact
     attributes:
-      label: Twitter / LinkedIn details 
+      label: Twitter / LinkedIn details
       description: We announce new features on Twitter + LinkedIn. When this is announced, and you'd like a mention, we'll gladly shout you out!
       placeholder: ex. @krrish_dh / https://www.linkedin.com/in/krish-d/
     validations:

diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
--- a/.github/pull_request_template.md
+++ b/.github/pull_request_template.md
@@ -1,11 +1,67 @@
+<!--
+👋 Hi there — please read before submitting.
+
+To keep the review queue healthy for everyone, **every external PR is
+auto-triaged** by an LLM bot ("Agent Shin") on open / reopen, regardless of
+whether the PR is a draft or marked ready for review. PRs that don't meet
+the rubric below are auto-closed with an explanation.
+
+To pass triage, your PR must satisfy AT LEAST ONE of:
+
+  (A) Link a related GitHub issue (e.g. "Fixes #1234" or "Resolves
+      https://github.com/BerriAI/litellm/issues/1234"), OR
+
+  (B) Provide ALL of the following IN THIS PR DESCRIPTION:
+      - A clear problem description (what bug or missing feature this addresses)
+      - Expected vs. actual behavior
+      - Visual QA proof (before/after screenshots, screen recording, or
+        terminal output demonstrating that the fix/feature works end-to-end)
+
+Every external PR (including drafts, regardless of age) also receives a
+Greptile code review. Any PR with a Greptile Confidence Score below 4/5 is
+auto-closed.
+
+If your PR was auto-closed and you've addressed the feedback, you have two
+options to bring it back:
+
+  - **Open a new PR** with the updated branch (recommended — GitHub does not
+    let external contributors reopen a PR that was closed by a bot or
+    maintainer).
+  - **Or** comment `@agent-shin reconsider` on the closed PR. Agent Shin
+    will re-run triage and reopen the PR if it now meets the bar.
+
+Internal BerriAI contributors are exempt from this auto-triage — fill in the
+Linear ticket section instead.
+-->
+
 ## Relevant issues
 
-<!-- e.g. "Fixes #000" -->
+<!-- e.g. "Fixes #000". If you have no related issue, fill in the
+"Problem description / Expected vs. Actual / QA proof" sections below. -->
 
 ## Linear ticket
 
-<!-- if you are an internal contributor, add the Linear ticket e.g. "Resolves LIT-1234" to magically link the Linear ticket to the GitHub PR -->
+<!-- INTERNAL CONTRIBUTORS ONLY: add the Linear ticket e.g. "Resolves LIT-1234"
+to magically link the Linear ticket to the GitHub PR. External contributors:
+leave this blank and fill in the problem/expected-actual/QA sections below. -->
 
+## Problem description
+
+<!-- What bug or missing feature does this PR address? One or two paragraphs.
+External contributors: required unless you linked a GitHub issue above. -->
+
+## Expected vs. actual behavior
+
+<!-- What did you expect to happen? What is happening today (before this PR)?
+External contributors: required unless you linked a GitHub issue above. -->
+
+## QA proof
+
+<!-- Required for external contributors: include before/after screenshots,
+a screen recording, or terminal/log output that demonstrates the fix or feature
+works end-to-end. For UI changes, before/after screenshots are mandatory.
+For backend changes, terminal output of a passing test or curl command is fine. -->
+
 ## Pre-Submission checklist
 
 **Please complete all items before asking a LiteLLM maintainer to review your PR**
@@ -13,7 +69,7 @@
 - [ ] I have Added testing in the [`tests/test_litellm/`](https://github.com/BerriAI/litellm/tree/main/tests/test_litellm) directory, **Adding at least 1 test is a hard requirement** - [see details](https://docs.litellm.ai/docs/extras/contributing_code)
 - [ ] My PR passes all unit tests on [`make test-unit`](https://docs.litellm.ai/docs/extras/contributing_code)
 - [ ] My PR's scope is as isolated as possible, it only solves 1 specific problem
-- [ ] I have requested a Greptile review by commenting `@greptileai` and received a **Confidence Score of at least 4/5** before requesting a maintainer review
+- [ ] I have received a Greptile **Confidence Score of at least 4/5** before requesting a maintainer review (Greptile reviews automatically on open; comment `@greptileai` to re-trigger after pushing fixes)
 
 ## Delays in PR merge?
 
@@ -36,13 +92,6 @@
 - [ ] **Merge / cherry-pick CI run**  
        Links:
 
-## Screenshots / Proof of Fix
-
-<!-- Include screenshots, screen recordings, or log output demonstrating that your changes work as expected.
-     For bug fixes: show reproduction before the fix and passing behavior after.
-     For new features: show the feature working end-to-end.
-     For UI changes: include before/after screenshots. -->
-
 ## Type
 
 <!-- Select the type of Pull Request -->

diff --git a/.github/scripts/close_low_quality_prs.py b/.github/scripts/close_low_quality_prs.py
new file mode 100644
--- /dev/null
+++ b/.github/scripts/close_low_quality_prs.py
@@ -1,0 +1,467 @@
+#!/usr/bin/env python3
+"""
+Auto-close low-quality pull requests.
+
+Closes open PRs (including drafts, regardless of age) that satisfy ALL of:
+  1. Have a Greptile (`greptile-apps`) review comment whose latest
+     "Confidence Score: X/5" is below the configured threshold (default: 4).
+  2. Are authored by an external OSS contributor (internal BerriAI
+     contributors are exempt).
+  3. Do not carry an opt-out label (default: "do not close").
+
+`--min-age-days` is retained as an opt-in safety net for one-off backfill
+runs (default: 0). The team's intent is that the count of open PRs equals
+the count of PRs internal collaborators need to action on, so neither age
+nor draft status acts as a free pass.
+
+For each match, the script posts an explanatory comment and closes the PR.
+Because OSS contributors *cannot* reopen a PR closed by the bot/maintainer
+(GitHub limitation), the close-comment instructs them to push their fixes
+and **open a fresh PR**, or to comment `@agent-shin reconsider` on the
+closed PR to have the LLM judge re-evaluate (and reopen on pass).
+
+Requires the `gh` CLI to be authenticated.
+
+Usage examples:
+    # Dry run (default) - prints what would be closed
+    python3 close_low_quality_prs.py
+
+    # Actually close matching PRs
+    python3 close_low_quality_prs.py --close
+
+    # Restrict to PRs at least N days old (one-off backfill safety net)
+    python3 close_low_quality_prs.py --min-age-days 7 --min-score 4 --close
+"""
+
+from __future__ import annotations
+
+import argparse
+import datetime as dt
+import json
+import re
+import subprocess
+import sys
+from pathlib import Path
+from typing import Iterable
+
+# Share constants with the sibling Agent Shin script instead of duplicating
+# them. `AGENT_SHIN_AUTO_CLOSE_MARKER` is the literal phrase the reconsider
+# provenance check keys off, and `INTERNAL_ASSOCIATIONS` is the exempt-author
+# set; drift between the two files would silently break reconsider for
+# Greptile-closed PRs (marker) or let one script close a PR the other would
+# skip (associations), with no test catching it.
+_SCRIPTS_DIR = Path(__file__).resolve().parent
+if str(_SCRIPTS_DIR) not in sys.path:
+    sys.path.insert(0, str(_SCRIPTS_DIR))
+from triage_with_llm import (  # noqa: E402
+    AGENT_SHIN_AUTO_CLOSE_MARKER,
+    INTERNAL_ASSOCIATIONS,
+)
+
+# Greptile's GitHub App appears as `greptile-apps[bot]` in REST API comments
+# and `greptile-apps` in `gh pr view --json` output. Accept either form.
+GREPTILE_BOT_LOGINS = frozenset({"greptile-apps", "greptile-apps[bot]"})
+
+# Matches lines like:
+#   <h3>Confidence Score: 3/5</h3>
+#   **Confidence Score: 4/5**
+#   Confidence Score: 5 / 5
+SCORE_PATTERN = re.compile(
+    r"confidence\s*score\s*[:\-]?\s*(\d+)\s*/\s*5",
+    re.IGNORECASE,
+)
+
+# Default labels that exempt a PR from auto-close. Defined at module scope (not
+# as a mutable argparse default) so that `--optout-label foo` REPLACES the
+# defaults instead of appending to them — the argparse `action="append"` +
+# `default=[...]` combination silently mutates the shared default list.
+DEFAULT_OPTOUT_LABELS = ("do not close", "keep open", "wip")
+
+
+def gh(*args: str) -> str:
+    """Run a `gh` CLI command and return stdout. Raises on non-zero exit."""
+    result = subprocess.run(
+        ["gh", *args],
+        capture_output=True,
+        text=True,
+        check=True,
+    )
+    return result.stdout
+
+
+# `gh pr list --limit` caps at 1000 (the CLI's documented hard ceiling).
+# Surface a warning if we ever hit that cap so the silent truncation is
+# visible in workflow logs instead of just being a missed close.
+GH_PR_LIST_LIMIT = 1000
+
+
+def fetch_open_prs(repo: str | None) -> list[dict]:
+    """Fetch all open PRs (number, createdAt, isDraft, labels, author).
+
+    Includes drafts: `gh pr list --state open` returns both ready-for-review
+    and draft PRs by default. This is the desired behavior — drafts are not
+    a free pass; the internal-collaborator open-PR queue should reflect every
+    PR that needs human attention regardless of draft status.
+    """
+    repo_args = ["--repo", repo] if repo else []
+    fields = "number,title,createdAt,isDraft,labels,author,url"
+    raw = gh(
+        "pr",
+        "list",
+        "--state",
+        "open",
+        "--limit",
+        str(GH_PR_LIST_LIMIT),
+        "--json",
+        fields,
+        *repo_args,
+    )
+    prs = json.loads(raw)
+    if len(prs) >= GH_PR_LIST_LIMIT:
+        # `gh pr list --limit N` returns at most N rows even if more exist;
+        # log a GitHub Actions warning so the truncation isn't silent.
+        message = (
+            f"fetch_open_prs hit the gh CLI cap ({GH_PR_LIST_LIMIT}); "
+            "the open-PR list is likely truncated. Switch to paginated "
+            "`gh api` calls if the repo regularly exceeds this cap."
+        )
+        print(f"::warning::{message}", file=sys.stderr)
+    return prs
+
+
+def fetch_pr_author_association(pr_number: int, repo: str | None) -> str:
+    """Return the GitHub `author_association` for a PR, uppercase.
+
+    Values: OWNER, MEMBER, COLLABORATOR, CONTRIBUTOR, FIRST_TIME_CONTRIBUTOR,
+    FIRST_TIMER, MANNEQUIN, NONE. Returns "" on lookup failure.
+    """
+    endpoint = (
+        f"repos/{repo}/pulls/{pr_number}"
+        if repo
+        else f"repos/{{owner}}/{{repo}}/pulls/{pr_number}"
+    )
+    try:
+        data = json.loads(gh("api", endpoint))
+    except subprocess.CalledProcessError:
+        return ""
+    return (data.get("author_association") or "").upper()
+
+
+def is_external_pr_author(pr: dict, repo: str | None) -> bool:
+    """Return True if the PR author is an external OSS contributor.
+
+    Internal = `OWNER` / `MEMBER` / `COLLABORATOR` association, or a bot login.
+    """
+    login = ((pr.get("author") or {}).get("login") or "").lower()
+    if login.endswith("[bot]") or login in {"dependabot", "github-actions"}:
+        return False
+    association = fetch_pr_author_association(pr["number"], repo)
+    # Fail-safe: if the API lookup failed (empty string), treat the author as
+    # internal so we don't auto-close their PR. Auto-close is destructive, so
+    # an unknown association should never make a PR eligible for closing.
+    if not association or association in INTERNAL_ASSOCIATIONS:
+        return False
+    return True
+
+
+def fetch_pr_comments(pr_number: int, repo: str | None) -> list[dict]:
+    """Fetch issue-level comments on a PR (where Greptile posts its summary).
+
+    Returns [] on API failure so a transient hiccup on any single PR doesn't
+    abort the whole daily sweep mid-loop. Matches the fail-safe pattern in
+    `fetch_pr_author_association`; downstream the empty list becomes a
+    `skip-no-greptile-score` action and the PR is re-evaluated on the next run.
+    """
+    endpoint = (
+        f"repos/{repo}/issues/{pr_number}/comments?per_page=100"
+        if repo
+        else f"repos/{{owner}}/{{repo}}/issues/{pr_number}/comments?per_page=100"
+    )
+    try:
+        raw = gh("api", "--paginate", endpoint)
+    except subprocess.CalledProcessError:
+        return []
+    comments: list[dict] = []
+    for line in raw.strip().splitlines():
+        line = line.strip()
+        if not line:
+            continue
+        try:
+            parsed = json.loads(line)
+        except json.JSONDecodeError:
+            return []
+        if isinstance(parsed, list):
+            comments.extend(parsed)
+        else:
+            comments.append(parsed)
+    return comments
+
+
+def extract_greptile_score(comments: Iterable[dict]) -> tuple[int, dict] | None:
+    """Return (score, comment) for the most recent Greptile-authored comment
+    that contains a "Confidence Score: X/5". Returns None if no such comment.
+
+    "Most recent" is determined by the comment's `updated_at` (falling back to
+    `created_at`), so re-reviews override earlier passes.
+    """
+    candidates: list[tuple[str, int, dict]] = []
+    for comment in comments:
+        user = (comment.get("user") or {}).get("login", "")
+        if user not in GREPTILE_BOT_LOGINS:
+            continue
+        body = comment.get("body") or ""
+        match = SCORE_PATTERN.search(body)
+        if not match:
+            continue
+        score = int(match.group(1))
+        timestamp = comment.get("updated_at") or comment.get("created_at") or ""
+        candidates.append((timestamp, score, comment))
+
+    if not candidates:
+        return None
+
+    candidates.sort(key=lambda triple: triple[0])
+    _, score, comment = candidates[-1]
+    return score, comment
+
+
+def parse_iso8601(value: str) -> dt.datetime:
+    """Parse a GitHub ISO-8601 timestamp into a timezone-aware datetime."""
+    return dt.datetime.fromisoformat(value.replace("Z", "+00:00"))
+
+
+def has_optout_label(pr: dict, optout_labels: set[str]) -> bool:
+    labels = {label.get("name", "").lower() for label in pr.get("labels", [])}
+    return bool(labels & {lbl.lower() for lbl in optout_labels})
+
+
+def close_pr(
+    pr: dict,
+    score: int,
+    threshold: int,
+    age_days: int,
+    repo: str | None,
+    dry_run: bool,
+    label: str | None,
+) -> None:
+    """Post the explanatory comment and close the PR."""
+    pr_number = pr["number"]
+    repo_args = ["--repo", repo] if repo else []
+
+    if dry_run:
+        print(
+            f"  [DRY RUN] Would close PR #{pr_number} "
+            f"(age={age_days}d, greptile={score}/5): {pr['title']}"
+        )
+        return
+
+    comment_body = (
+        f"👋 Hi, thanks for the PR! {AGENT_SHIN_AUTO_CLOSE_MARKER}, the automated triage "
+        "bot for this repository. Closing as part of automated PR triage.\n\n"
+        f"Greptile's most recent review scored this PR **{score}/5**, below "
+        f"our merge bar of **{threshold}/5**.\n\n"
+        "We close low-confidence PRs aggressively to keep the review queue "
+        "manageable for maintainers and contributors alike. **This is not a "
+        "rejection of the idea** — to bring this back:\n\n"
+        "1. Push the fixes that address Greptile's feedback (continue using "
+        "your existing branch is fine).\n"
+        "2. **Open a new PR** with the updated branch. Greptile will review "
+        "it again, and if it scores "
+        f"**{threshold}/5 or higher** a maintainer will take another look.\n\n"
+        "_Why open a new PR instead of reopening this one?_ GitHub does not "
+        "let external contributors reopen a PR that was closed by a bot or "
+        "maintainer, so a fresh PR is the most reliable path forward. If you "
+        "would prefer this exact PR re-evaluated, comment "
+        "`@agent-shin reconsider` once you've pushed the fixes — Agent Shin "
+        "will re-run triage and reopen this PR if it now meets the bar.\n\n"
+        "Thanks for contributing to LiteLLM. We know auto-closures can sting; "
+        "the goal is to keep the project healthy, not to dismiss your work."
+    )
+    gh("pr", "comment", str(pr_number), "--body", comment_body, *repo_args)
+
+    if label:
+        try:
+            gh("pr", "edit", str(pr_number), "--add-label", label, *repo_args)
+        except subprocess.CalledProcessError as exc:
+            stderr = (exc.stderr or "").strip()
+            print(f"  warn: failed to add label '{label}' to #{pr_number}: {stderr}")
+
+    gh("pr", "close", str(pr_number), *repo_args)
+    print(f"  Closed PR #{pr_number} (greptile={score}/5, age={age_days}d)")
+
+
+def evaluate_pr(
+    pr: dict,
+    now: dt.datetime,
+    min_age_days: int,
+    min_score: int,
+    repo: str | None,
+    optout_labels: set[str],
+) -> tuple[str, int | None, int | None]:
+    """Decide whether to close `pr`.
+
+    Returns (action, score_or_none, age_days_or_none) where action is one of:
+        "skip-too-young", "skip-optout-label", "skip-internal",
+        "skip-no-greptile-score", "skip-score-ok", or "close".
+
+    Drafts are NOT skipped — the goal is "open PR count == PRs internal
+    collaborators need to action on", and a draft that Greptile scored <4/5
+    is still in that queue. Authors can opt out via the `wip` label (see
+    `DEFAULT_OPTOUT_LABELS`) if they need to keep a long-lived draft open.
+    """
+    if has_optout_label(pr, optout_labels):
+        return ("skip-optout-label", None, None)
+
+    created = parse_iso8601(pr["createdAt"])
+    age_days = (now - created).days
+    # `min_age_days` defaults to 0 (close as soon as Greptile scores low).
+    # Set a positive value via --min-age-days for one-off backfill runs that
+    # want to skip very-young PRs.
+    if min_age_days > 0 and age_days < min_age_days:
+        return ("skip-too-young", None, age_days)
+
+    # Only auto-close external OSS contributors. Internal contributors
+    # (BerriAI org members) handle their own backlog.
+    if not is_external_pr_author(pr, repo):
+        return ("skip-internal", None, age_days)
+
+    comments = fetch_pr_comments(pr["number"], repo)
+    extraction = extract_greptile_score(comments)
+    if extraction is None:
+        return ("skip-no-greptile-score", None, age_days)
+
+    score, _ = extraction
+    if score >= min_score:
+        return ("skip-score-ok", score, age_days)
+
+    return ("close", score, age_days)
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "--repo",
+        type=str,
+        default=None,
+        help="Repository (owner/repo). Auto-detected if omitted.",
+    )
+    parser.add_argument(
+        "--min-age-days",
+        type=int,
+        default=0,
+        help=(
+            "Minimum age (in days) before a PR is eligible. Default 0 = "
+            "close as soon as Greptile flags it. Set a positive value for "
+            "one-off backfill runs that want to spare very-young PRs."
+        ),
+    )
+    parser.add_argument(
+        "--min-score",
+        type=int,
+        default=4,
+        choices=range(1, 6),
+        help="Greptile score below which a PR is closed (default: 4 -> closes <4/5).",
+    )
+    parser.add_argument(
+        "--optout-label",
+        action="append",
+        default=None,
+        help=(
+            "Label(s) that exempt a PR from auto-close. Repeat to add more. "
+            "Case-insensitive. When omitted, defaults to "
+            f"{list(DEFAULT_OPTOUT_LABELS)!r}; passing this flag REPLACES the "
+            "defaults (argparse `append` with a mutable default would append "
+            "instead, which we explicitly avoid)."
+        ),
+    )
+    parser.add_argument(
+        "--close-label",
+        type=str,
+        default=None,
+        help=(
+            "Optional label to add to PRs that get auto-closed "
+            "(e.g. 'auto-closed-low-quality'). Must already exist on the repo."
+        ),
+    )
+    parser.add_argument(
+        "--close",
+        action="store_true",
+        help="Actually close matching PRs (default is dry-run).",
+    )
+    parser.add_argument(
+        "--limit",
+        type=int,
+        default=None,
+        help=(
+            "Maximum number of PRs to close in one run (safety net). "
+            "Applied in dry-run too so `--limit N` previews exactly the "
+            "first N closures."
+        ),
+    )
+    args = parser.parse_args()
+
+    dry_run = not args.close
+    if dry_run:
+        print("=== DRY RUN MODE (pass --close to actually close PRs) ===\n")
+
+    print("Fetching open PRs...")
+    prs = fetch_open_prs(args.repo)
+    print(f"Found {len(prs)} open PRs.\n")
+
+    now = dt.datetime.now(dt.timezone.utc)
+    optout_labels = set(args.optout_label or DEFAULT_OPTOUT_LABELS)
+
+    closed = 0
+    summary = {
+        "close": 0,
+        "skip-too-young": 0,
+        "skip-optout-label": 0,
+        "skip-internal": 0,
+        "skip-no-greptile-score": 0,
+        "skip-score-ok": 0,
+    }
+
+    for pr in sorted(prs, key=lambda p: p["createdAt"]):
+        action, score, age_days = evaluate_pr(
+            pr,
+            now,
+            args.min_age_days,
+            args.min_score,
+            args.repo,
+            optout_labels,
+        )
+        summary[action] = summary.get(action, 0) + 1
+
+        if action != "close":
+            continue
+
+        assert score is not None and age_days is not None
+        print(
+            f"#{pr['number']}: \"{pr['title']}\" "
+            f"(age={age_days}d, greptile={score}/5) -> close"
+        )
+        close_pr(
+            pr,
+            score=score,
+            threshold=args.min_score,
+            age_days=age_days,
+            repo=args.repo,
+            dry_run=dry_run,
+            label=args.close_label,
+        )
+
+        closed += 1
+        if args.limit is not None and closed >= args.limit:
+            verb = "Would close" if dry_run else "Closed"
+            print(f"\nReached --limit={args.limit} ({verb} count); stopping.")
+            break
+
+    print("\n=== Summary ===")
+    for key, value in summary.items():
+        print(f"  {key:28s} {value}")
+    print(f"\nTotal {'would close' if dry_run else 'closed'}: {summary['close']}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

diff --git a/.github/scripts/triage_with_llm.py b/.github/scripts/triage_with_llm.py
new file mode 100644
--- /dev/null
+++ b/.github/scripts/triage_with_llm.py
@@ -1,0 +1,927 @@
+#!/usr/bin/env python3
+"""
+Agent Shin — LLM-as-judge triage for external OSS pull requests and issues.
+
+Evaluates a single PR or issue against the contribution rubric and, when the
+LLM judge marks it as failing, posts an explanatory comment + closes the
+PR/issue. Re-triggers on `reopened` so contributors can iterate back in by
+filling in the missing pieces and reopening.
+
+Internal BerriAI contributors (`author_association` in {OWNER, MEMBER,
+COLLABORATOR}) and bot accounts are skipped entirely.
+
+Usage:
+    triage_with_llm.py --repo owner/repo --pr 1234
+    triage_with_llm.py --repo owner/repo --issue 5678
+    triage_with_llm.py --repo owner/repo --pr 1234 --close    # actually close
+    triage_with_llm.py --repo owner/repo --pr 1234 --print-prompt  # show prompt
+
+Defaults are SAFE: without `--close` the script writes a verdict to stdout (and,
+when running in GitHub Actions, to $GITHUB_STEP_SUMMARY) but takes no GitHub
+write actions.
+
+Environment:
+    GH_TOKEN / GITHUB_TOKEN  - for `gh` CLI auth (auto-set in Actions)
+    OPENAI_API_KEY           - required when --close is passed
+    OPENAI_BASE_URL          - optional (route to any OpenAI-compatible API)
+    TRIAGE_MODEL             - optional model override (default: gpt-5.4-mini)
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import subprocess
+import sys
+import textwrap
+from typing import Any
+
+DEFAULT_MODEL = "gpt-5.4-mini"
+
+INTERNAL_ASSOCIATIONS = frozenset({"OWNER", "MEMBER", "COLLABORATOR"})
+
+# Marker phrase Agent Shin always includes in its auto-close comments
+# (see `format_pr_close_comment` / `format_issue_close_comment`). The
+# provenance check for reconsider matches this marker against a comment
+# authored by the same bot login that performed the most recent `closed`
+# event, so a contributor cannot reopen a PR/issue that a maintainer
+# closed after a prior Agent Shin auto-close. Keep the marker in sync
+# with the literal text in those formatter functions.
+AGENT_SHIN_AUTO_CLOSE_MARKER = "I'm **Agent Shin**"
+
+# Model families that require `reasoning_effort` to be set, and that reject
+# `temperature != 1` unless `reasoning_effort` is "none". For these models we
+# pass `reasoning_effort="none"` so a `temperature=0` deterministic judgment
+# is still accepted. See litellm/llms/openai/chat/gpt_5_transformation.py for
+# the full set of constraints LiteLLM applies to these models.
+GPT5_FAMILY_PREFIX = "gpt-5"
+
+# Regexes for picking off "obvious passes" without burning LLM tokens.
+#
+# Keep this list to GitHub's documented PR-closing keywords only
+# (https://docs.github.com/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue).
+# Casual mentions like "see #1234" or "ref #1234" are intentionally NOT
+# auto-passed — they should fall through to the LLM judge, which has the
+# stricter rubric "a bare issue number without a closing keyword counts only
... diff truncated: showing 800 of 4242 lines

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit 7b366fe. Configure here.}

[mateo-berri]

@greptileai

[mateo-berri]

Following up on the one remaining note in the Greptile summary — "tests/local_testing/test_custom_callback_input.py and tests/local_testing/test_stream_chunk_builder.py are modified without explanation in a triage-scoped PR; worth a quick check that they were intentionally included."

Confirming intentional, and leaving them in this PR. Reasoning:

• The two 6-line diffs are a strict bug-fix over a real silent-exception-swallowing pattern that exists on litellm_internal_staging:

  except Exception as e:
      if "openai-internal" in str(e):
          pytest.skip("Skipping test due to openai-internal error")
  # any non-"openai-internal" exception is silently dropped; the next line
  # then runs with an unbound `response` / `completion` and fails with an
  # `UnboundLocalError` that masks the real upstream cause.

  This PR adds (a) a pytest.skip for model_not_found / does not exist (gpt-4o-audio-preview was retired upstream and was reliably 404-ing in CI as of 2026-05-19), and (b) a raise for any unknown exception so the test reports the real cause instead of UnboundLocalError. The pattern is identical to the previously-merged #28191 (fix(tests): migrate realtime + rerank tests off shut-down upstream models), which was the precedent reference in the commit message of 1845b0d.

• The summary itself reaches the same conclusion in two places: "The two unrelated test file changes are minor and correct" and, in the Important Files table, "Adds model-unavailability skip conditions and a previously-missing raise; fixes a silent exception-swallowing bug." The only finding left is the scoping note.

• Reverting the two files on this branch to make the PR "triage-only" would re-introduce the silent-swallow bug + leave CI red on gpt-4o-audio-preview — strictly worse for the codebase than keeping the fix. Splitting them into a separate PR would be churn for a 6-line strict-improvement diff that already ships with its own commit (1845b0d) and commit-message rationale.

Leaving the two test files as-is.

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.