Skip to content
Merged
Show file tree
Hide file tree
Changes from 49 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
2e7fdc6
feat(triage): auto-close stale PRs with Greptile score <4/5
cursoragent May 17, 2026
ac18f14
docs(templates): require expected/actual + QA proof for external cont…
cursoragent May 17, 2026
7b4a093
feat(triage): Agent Shin LLM-as-judge for external PRs and issues
cursoragent May 17, 2026
483042e
feat(triage): scope Greptile auto-closer to external contributors + d…
cursoragent May 17, 2026
4961bf4
fix(workflows): scheduled cron closes PRs; safe --close strip in triage
cursoragent May 17, 2026
4014333
fix(triage): scheduled cron stays dry-run; dedent prompts before inte…
cursoragent May 17, 2026
edddf0c
Fix bugs in auto-close PR triage scripts
cursoragent May 17, 2026
09da8a7
feat(triage): default Agent Shin to gpt-5.4-mini with reasoning_effor…
cursoragent May 17, 2026
c501f74
fix(triage): bugbot — drop dead gh_json and fix --optout-label append…
cursoragent May 17, 2026
1ac5bee
fix(triage): bugbot — tighten linked-issue regex, fail-safe author_as…
cursoragent May 17, 2026
638dba9
fix(workflows): bugbot — gate Agent Shin --close on '= true' not '!= …
cursoragent May 17, 2026
ff57d5b
feat(triage): close any PR (incl. drafts, any age); add @agent-shin r…
cursoragent May 18, 2026
420547f
test(triage): pin safe behavior for curly braces in PR/issue title+body
cursoragent May 18, 2026
4e0968a
fix(triage): bugbot — reconsider dry-run + bot-closed guard + rate limit
cursoragent May 18, 2026
f876173
feat(triage): 1-day grace period before close + SwiftWinds immediate-…
cursoragent May 19, 2026
e0eeb73
fix: skip grace-period text in close comment for IMMEDIATE_CLOSE_LOGINS
cursoragent May 19, 2026
b0fc8c2
fix(close-low-quality-prs): report actual closes in dry-run summary
cursoragent May 19, 2026
0ad6836
chore(triage): vendor Agent Shin (#28117) onto demo branch
claude May 24, 2026
b4d9703
feat(triage): add "ready for review" label lifecycle to Agent Shin
claude May 24, 2026
7fb1ec6
Merge PR #28117 + #28147 (Agent Shin auto-close + reconsider safety)
May 25, 2026
b7ed45e
Merge PR #28758 — keep #28147 versions of overlapping triage files; r…
May 25, 2026
125f104
Port review-gate feature from #28758 onto #28147 triage scripts
May 25, 2026
8b5913b
agent_shin: extract shared constants/helpers; cover review_gate.yml i…
cursoragent May 25, 2026
1d0d490
agent_shin: fix bug bundle (gated LLM key, author-filtered marker ded…
cursoragent May 25, 2026
19dfdd4
agent_shin: fix review_gate close-after-regression and case-insensiti…
cursoragent May 25, 2026
9e41a22
feat(triage): add one-shot 7-day heads-up sweep for Agent Shin rollout
mateo-berri May 26, 2026
88543ed
fix: gate reconsider workflow OPENAI_API_KEY + remove dead actions wr…
cursoragent May 26, 2026
234471a
fix: address triage script edge cases
cursoragent May 26, 2026
09720a7
fix(scripts): per-PR error isolation and limit grace warnings in clos…
cursoragent May 26, 2026
2c49dcf
fix(agent-shin): remove 1000-PR cap on bulk sweeps; sweep entire backlog
mateo-berri May 28, 2026
c4d74ee
fix(agent-shin): require non-mocked end-to-end QA proof for PR pass
mateo-berri May 28, 2026
9f4b795
feat(agent-shin): link blog explainer from every action-required bot …
mateo-berri May 28, 2026
45fc4c9
fix(review_gate): raise sweep limit from 1000 to 100000 to match GH_L…
cursoragent May 28, 2026
e21fe21
Fix three Agent Shin triage edge cases
cursoragent May 28, 2026
dd02b76
fix: pass min_greptile_score through to ready-for-review comment text
cursoragent May 28, 2026
f7e514a
feat(agent-shin): warmer triage comments — bullet-train emoji, 'what …
mateo-berri May 28, 2026
dc93e10
Merge remote-tracking branch 'origin/litellm_internal_staging' into l…
mateo-berri Jun 14, 2026
9138060
feat(agent-shin): gate triage on a dogfood allowlist
mateo-berri Jun 17, 2026
1e069a8
feat(agent-shin): tighten QA-proof and issue rubrics, ack reconsider …
mateo-berri Jun 15, 2026
bdae163
fix(agent-shin): shorten auto-close grace to 2 hours and drop the ins…
mateo-berri Jun 17, 2026
f8ece2b
fix(agent-shin): gate reconsider reopen on an Agent Shin close marker
mateo-berri Jun 17, 2026
f73719f
fix(agent-shin): stamp close marker on sweep closes and disclose regr…
mateo-berri Jun 17, 2026
69c2e7b
fix(triage): tighten Agent Shin reconsider reopen guards
cursoragent Jun 17, 2026
5434b21
style(agent-shin): black-format reconsider guard hardening
mateo-berri Jun 17, 2026
a0d4edf
docs(agent-shin): scope dry-run wrapper docstring to the single exist…
mateo-berri Jun 17, 2026
e8fb7b2
Merge remote-tracking branch 'origin/litellm_internal_staging' into l…
mateo-berri Jun 17, 2026
7e99f48
chore(agent-shin): defer issue/PR template changes to the rollout PR
mateo-berri Jun 18, 2026
428626c
Merge remote-tracking branch 'origin/litellm_internal_staging' into l…
mateo-berri Jun 18, 2026
5791024
ci(agent-shin): hash-pin the openai install in privileged triage work…
mateo-berri Jun 18, 2026
8d38de0
ci(agent-shin): gate rollout heads-up real run behind manual dispatch
mateo-berri Jun 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions .github/scripts/_agent_shin_actions.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
"""Dry-run wrapper(s) around Agent Shin GitHub mutations.

The rollout scripts currently need only one mutation wrapped, so this module
exposes a single ``maybe_post_comment`` helper. It takes a ``dry_run: bool``
keyword argument and the body is intentionally trivial:

if dry_run:
print(...) # log what we would do, return
return
real_mutation(...) # otherwise, actually do it

That shape means a dry-run preview differs from the real run in exactly one
line per side effect: the call site. So when you `python3 script.py` locally
without ``--close``, you can be confident the actions printed are the ones the
GitHub Action would have performed (modulo ordering on retry/error paths,
which are deliberately simple). Any further mutation a rollout script needs
should get the same ``maybe_*`` treatment instead of calling the raw
``triage_with_llm`` mutation directly.

Importing from this module pulls in the real mutation from ``triage_with_llm``
— call sites in the rollout scripts should NEVER import ``post_comment``
directly; that would skip the dry-run gate and is the bug class this module
exists to prevent.
"""

from __future__ import annotations

import sys
import textwrap

# Import the module itself rather than the bare names so monkeypatching
# `triage_with_llm.post_comment` (or any of the other mutations) in tests is
# reflected here — `from triage_with_llm import post_comment` would bind the
# original function to a local name and bypass the patch, defeating the whole
# point of these wrappers.
import triage_with_llm


def _log(line: str) -> None:
"""Print a single dry-run line to stdout (one log statement per side effect)."""
print(line, file=sys.stdout, flush=True)


def maybe_post_comment(repo: str, number: int, body: str, *, dry_run: bool) -> None:
"""Post a comment on ``repo#number`` — or, in dry-run, log what we would post."""
if dry_run:
_log(f"[DRY RUN] comment {repo}#{number}:")
_log(textwrap.indent(body, " "))
return
triage_with_llm.post_comment(repo, number, body)
211 changes: 211 additions & 0 deletions .github/scripts/agent_shin_shared.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,211 @@
"""Constants and helpers shared by Agent Shin's triage scripts.

Both `triage_with_llm.py` (the LLM-judge entrypoint) and
`close_low_quality_prs.py` (the daily Greptile-score sweep) need to
agree on the same notions of:

* What counts as a Greptile-authored review comment
(``GREPTILE_BOT_LOGINS``) and how to extract a confidence score from
its body (``SCORE_PATTERN`` / :func:`extract_greptile_score`).
* How long the 2-hour grace window is (``GRACE_PERIOD_SECONDS``) and
the HTML marker stamped into a grace-warning comment so the *other*
script can see "Agent Shin already warned" and behave accordingly
(``GRACE_COMMENT_MARKER``).
* Who Agent Shin is on GitHub (``AGENT_SHIN_DEFAULT_BOT_LOGIN``).
* How GitHub-style ISO-8601 timestamps round-trip into timezone-aware
:class:`datetime.datetime` (:func:`parse_iso8601`).

Keeping these in one module means a future change (new Greptile output
format, a longer grace window, a new allowlisted account) is a single edit
instead of two — the original split version had to call out in comments
that the two copies "must stay in sync" precisely because nothing
enforced it.
"""

from __future__ import annotations

import datetime as dt
import json
import os
import re
import subprocess
from typing import Iterable

GREPTILE_BOT_LOGINS = frozenset({"greptile-apps", "greptile-apps[bot]"})

SCORE_PATTERN = re.compile(
r"confidence\s*score\s*[:\-]?\s*(\d+)\s*/\s*5",
re.IGNORECASE,
)

GRACE_COMMENT_MARKER = "<!-- agent-shin:grace-warning -->"

# Hidden HTML marker stamped on every Agent Shin auto-close comment (the LLM
# judge's grace/review-gate close and the daily Greptile sweep's close).
# `was_closed_by_agent_shin` requires this marker — not just the closing actor —
# before `@agent-shin reconsider` may reopen, because the `github-actions[bot]`
# identity is shared with every other workflow in the repo and is not unique to
# Agent Shin. Both close paths must stamp it or the reconsider path silently
# rejects the contributor.
AGENT_SHIN_CLOSE_MARKER = "<!-- agent-shin:closed -->"

# 2 hours between the grace warning and the auto-close. Short enough to
# dogfood the "fix it before it closes" loop in one sitting; bump back up
# (e.g. 86400 for a day) for the public rollout.
GRACE_PERIOD_SECONDS = 7200

AGENT_SHIN_DEFAULT_BOT_LOGIN = "github-actions[bot]"


def _logins(*names: str) -> frozenset[str]:
"""Build a login set normalized for case-insensitive membership checks.

Callers compare via ``login.lower() in <set>``, so the stored values
must be lowercase. Normalizing here lets the literals keep each
account's canonical GitHub casing (e.g. ``SwiftWinds``) for
readability without breaking the lookup.
"""
return frozenset(name.lower() for name in names)


# Dogfood rollout gate. While this set is non-empty, Agent Shin acts ONLY on
# PRs/issues authored by these logins and skips everyone else. For an
# allowlisted author the usual internal/external classification is bypassed, so
# an internal account (e.g. a maintainer's own work login) still gets triaged
# while the bot is being tested on a small set of accounts. Empty the set to
# lift the restriction and restore full triage for the public rollout. Logins
# are compared case-insensitively.
ALLOWLIST_LOGINS = _logins("mateo-berri", "SwiftWinds")

# `gh {pr,issue} list` has no "fetch everything" flag — `--limit` is the only
# control and it defaults to 30. Pass a ceiling far above any realistic open
# backlog (low thousands today) so gh paginates the API until the queue is
# exhausted rather than silently truncating. The bulk sweeps MUST see the whole
# backlog: gh lists newest-first, so a low cap drops the *oldest* PRs/issues —
# exactly the stale ones a low-quality sweep is meant to catch.
GH_LIST_ALL_LIMIT = 100_000


def extract_greptile_score(comments: Iterable[dict]) -> tuple[int, dict] | None:
"""Return (score, comment) for the most recent Greptile-authored comment
that contains a "Confidence Score: X/5". Returns None if no such comment.

"Most recent" is determined by the comment's `updated_at` (falling back to
`created_at`), so re-reviews override earlier passes.
"""
candidates: list[tuple[str, int, dict]] = []
for comment in comments:
user = (comment.get("user") or {}).get("login", "")
if user not in GREPTILE_BOT_LOGINS:
continue
body = comment.get("body") or ""
match = SCORE_PATTERN.search(body)
if not match:
continue
score = int(match.group(1))
timestamp = comment.get("updated_at") or comment.get("created_at") or ""
candidates.append((timestamp, score, comment))

if not candidates:
return None

candidates.sort(key=lambda triple: triple[0])
_, score, comment = candidates[-1]
return score, comment


def parse_iso8601(value: str) -> dt.datetime:
"""Parse a GitHub ISO-8601 timestamp into a timezone-aware datetime."""
return dt.datetime.fromisoformat(value.replace("Z", "+00:00"))


def gh(*args: str) -> str:
"""Run a `gh` CLI command and return stdout. Raises on non-zero exit.

Shared by both Agent Shin entrypoints so a future change here
(timeout handling, logging, retry on transient failures) only needs
to be made once.
"""
result = subprocess.run(
["gh", *args],
capture_output=True,
text=True,
check=True,
)
return result.stdout


def list_open_items(kind: str, *, repo: str | None, fields: str) -> list[dict]:
"""Return EVERY open PR (``kind="pr"``) or issue (``kind="issue"``) in ``repo``.

Wraps ``gh {pr,issue} list`` with ``--limit GH_LIST_ALL_LIMIT`` so the full
backlog is fetched instead of the default 30 (or any other arbitrary cap).
Both bulk sweeps — the daily Greptile closer and the one-shot rollout
heads-up — rely on this seeing the whole queue, including the oldest items.

``fields`` is the comma-separated ``--json`` field list the caller needs
(e.g. ``"number"`` for the rollout, the full set for the closer).
"""
if kind not in ("pr", "issue"):
raise ValueError(f"kind must be 'pr' or 'issue', got {kind!r}")
repo_args = ["--repo", repo] if repo else []
raw = gh(
kind,
"list",
"--state",
"open",
"--limit",
str(GH_LIST_ALL_LIMIT),
"--json",
fields,
*repo_args,
)
return json.loads(raw)


def seconds_since_latest_marker_comment(
comments: Iterable[dict],
*,
marker: str,
bot_login: str | None = None,
now: dt.datetime | None = None,
) -> float | None:
"""Return seconds since the bot's most recent comment containing ``marker``.

Filters comments by author so a contributor who quotes the HTML
marker (e.g. via GitHub's "Quote reply" feature, which preserves
HTML comments in the raw markdown of the quoted text) is not
mistaken for a bot warning — that would silently reset cooldown
timers and suppress legitimate notifications.

``bot_login`` defaults to the `AGENT_SHIN_BOT_LOGIN` env override or
``AGENT_SHIN_DEFAULT_BOT_LOGIN`` so callers normally don't need to
pass it. ``now`` is injectable for tests / callers (like the daily
sweep) that want every age calculation pinned to one snapshot.
"""
expected_login = (
bot_login
or os.environ.get("AGENT_SHIN_BOT_LOGIN")
or AGENT_SHIN_DEFAULT_BOT_LOGIN
).lower()
latest: dt.datetime | None = None
for comment in comments:
author = ((comment.get("user") or {}).get("login") or "").lower()
if author != expected_login:
continue
body = comment.get("body") or ""
if marker not in body:
continue
created = comment.get("created_at")
if not created:
continue
try:
ts = parse_iso8601(created)
except ValueError:
continue
if latest is None or ts > latest:
latest = ts
if latest is None:
return None
reference = now if now is not None else dt.datetime.now(dt.timezone.utc)
return (reference - latest).total_seconds()
Loading
Loading