Skip to content
Merged
9 changes: 9 additions & 0 deletions .github/workflows/pr-review.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,8 @@ jobs:
CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
# Copilot engine reuses GH_PAT (a separate user PAT with Copilot subscription).
COPILOT_GITHUB_TOKEN: ${{ secrets.GH_PAT }}
# Gemini engine auth — requires GOOGLE_API_KEY repo secret.
GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
# Default to dry-run. workflow_dispatch can override; repository_dispatch
# (mention) always runs live; vars.LIVE_MODE=true switches scheduled runs live.
DRY_RUN: ${{ github.event_name == 'repository_dispatch' && 'false' || inputs.dry_run || (vars.LIVE_MODE == 'true' && 'false' || 'true') }}
Expand Down Expand Up @@ -107,13 +109,20 @@ jobs:
case "$REVIEW_ENGINE" in
claude)
install_claude
# Best-effort install of fallbacks for duck/fallback support
npm install -g @google/gemini-cli || true
if ! env GH_TOKEN="$COPILOT_GITHUB_TOKEN" gh copilot --version > /dev/null 2>&1; then
echo "::warning::gh copilot built-in unavailable with user token — rate-limit fallback will be unavailable."
fi
Comment thread
don-petry marked this conversation as resolved.
;;
gemini)
npm install -g @google/gemini-cli
install_claude || true
;;
copilot)
gh extension install github/gh-copilot
Comment thread
don-petry marked this conversation as resolved.
Outdated
install_claude || true
npm install -g @google/gemini-cli || true
;;
*)
echo "::error::Unknown REVIEW_ENGINE=$REVIEW_ENGINE"
Expand Down
31 changes: 21 additions & 10 deletions AGENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ A scheduled GitHub Action that reviews open PRs on don-petry's behalf.
Runs hourly, classifies risk, auto-approves low/medium-risk PRs that pass all
quality gates, and escalates high-risk or gated PRs for human review.

Supports two LLM engines via the `REVIEW_ENGINE` repo variable: **Claude** (default)
Supports three LLM engines via the `REVIEW_ENGINE` repo variable: **Claude** (default), **Gemini**,
and **Copilot**.

## How it works
Expand Down Expand Up @@ -55,14 +55,14 @@ and **Copilot**.

### Engine model mapping

| Tier | Claude primary | Copilot primary |
|---|---|---|
| Triage | Haiku 4.5 | GPT-5-mini |
| Deep review | Sonnet 4.6 | GPT-5.2 |
| Rubber duck | GPT-5.4 (cross) | Sonnet 4.6 (cross) |
| Synthesis | Sonnet 4.6 | GPT-5.2 |
| Security audit | Opus 4.6 | GPT-5.4 |
| Action / single review | Sonnet 4.6 / Opus 4.6 | GPT-5.2 / GPT-5.4 |
| Tier | Claude primary | Gemini primary | Copilot primary |
|---|---|---|---|
| Triage | Haiku 4.5 | Gemini 2.0 Flash | GPT-5-mini |
| Deep review | Sonnet 4.6 | Gemini 1.5 Pro | GPT-5.2 |
| Rubber duck | GPT-5.4 (cross) | Sonnet 4.6 (cross) | Sonnet 4.6 (cross) |
| Synthesis | Sonnet 4.6 | Gemini 1.5 Pro | GPT-5.2 |
| Security audit | Opus 4.6 | Gemini 1.5 Pro | GPT-5.4 |
| Action / single review | Sonnet 4.6 / Opus 4.6 | Gemini 1.5 Pro | GPT-5.2 / GPT-5.4 |

**Cost profile:**
- ~80% of PRs: triage + single confirm (2 calls, ~30s)
Expand Down Expand Up @@ -168,6 +168,14 @@ Store as a repo secret:
gh secret set CLAUDE_CODE_OAUTH_TOKEN --repo don-petry/pr-review-agent
```

#### Gemini engine

Requires a Google API Key. Store as a repo secret:

```
gh secret set GOOGLE_API_KEY --repo don-petry/pr-review-agent
```

#### Copilot engine

Create a GitHub PAT with Copilot scope. Store as a repo secret:
Expand All @@ -179,6 +187,9 @@ gh secret set COPILOT_GITHUB_TOKEN --repo don-petry/pr-review-agent
### 4. Choose your engine

```
# Use Gemini:
gh variable set REVIEW_ENGINE --body gemini --repo don-petry/pr-review-agent

# Use Copilot (GPT models):
gh variable set REVIEW_ENGINE --body copilot --repo don-petry/pr-review-agent

Expand Down Expand Up @@ -225,7 +236,7 @@ gh workflow run pr-review.yml --repo don-petry/pr-review-agent -f dry_run=false

## Tuning

- **Review engine** — `REVIEW_ENGINE` repo variable: `claude` (default) or
- **Review engine** — `REVIEW_ENGINE` repo variable: `claude` (default), `gemini`, or
`copilot`. Controls which CLI and model family is used.
- **Risk rules** — edit `prompts/shared.md` (taxonomy), or the per-tier
prompts (`prompts/deep-review.md`, `prompts/security-audit.md`).
Expand Down
13 changes: 10 additions & 3 deletions IMPLEMENTATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,13 +166,19 @@ The `pr-review.yml` workflow includes fallback logic:

```bash
if [ "$rc" -eq 2 ] && [ "${REVIEW_ENGINE:-claude}" = "claude" ]; then
echo "Claude rate limit hit — switching to Copilot engine for remaining PRs"
echo "Claude rate limit hit — switching to Gemini engine for remaining PRs"
export REVIEW_ENGINE=gemini
bash scripts/review-one-pr.sh "$pr_url" || rc=$?
fi

if [ "$rc" -eq 2 ] && [ "${REVIEW_ENGINE}" = "gemini" ]; then
echo "Gemini rate limit hit — switching to Copilot engine for remaining PRs"
export REVIEW_ENGINE=copilot
bash scripts/review-one-pr.sh "$pr_url" || rc=$?
fi
```

If Claude hits rate limits, the workflow switches to GitHub Copilot for remaining PRs in the batch. This ensures reviews continue even under high load.
If Claude hits rate limits, the workflow switches to Gemini, and then to GitHub Copilot if needed. This ensures reviews continue even under high load.
Comment thread
don-petry marked this conversation as resolved.
Outdated

## Metrics and Monitoring

Expand All @@ -198,7 +204,7 @@ gh run view <run-id> --repo don-petry/pr-review-agent --log

| Variable | Default | Purpose |
|----------|---------|---------|
| `REVIEW_ENGINE` | `claude` | Primary review engine |
| `REVIEW_ENGINE` | `claude` | Primary review engine: `claude`, `gemini`, or `copilot` |
| `DRY_RUN` | `true` | If true, don't post reviews |
| `MAX_PRS` | `10` | Max reviews per run |
| `CANDIDATE_LIMIT` | `100` | Max candidates scanned |
Expand All @@ -210,6 +216,7 @@ gh run view <run-id> --repo don-petry/pr-review-agent --log
| Secret | Purpose |
|--------|---------|
| `CLAUDE_CODE_OAUTH_TOKEN` | Claude Code authentication |
| `GOOGLE_API_KEY` | Gemini API authentication |
| `DON_PETRY_BOT_PETRY_PROJECT_PAT` | Machine user PAT for GitHub API access |
| `COPILOT_GITHUB_TOKEN` | GitHub Copilot fallback token |
Comment thread
don-petry marked this conversation as resolved.
Outdated

Expand Down
3 changes: 2 additions & 1 deletion SETUP.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,15 @@ Store these in the repository settings (`Settings → Secrets and variables →
| Secret | Description | Source |
|--------|-------------|--------|
| `CLAUDE_CODE_OAUTH_TOKEN` | Claude Code authentication token | `claude setup-token` |
| `GOOGLE_API_KEY` | Gemini API authentication token | Google AI Studio / Vertex AI |
| `DON_PETRY_BOT_PETRY_PROJECT_PAT` | Machine user fine-grained PAT | Generated from machine user account settings |
| `COPILOT_GITHUB_TOKEN` | GitHub Copilot token (optional, for fallback) | GitHub PAT with Copilot scope |
Comment thread
don-petry marked this conversation as resolved.
Outdated

### Repository Variables (Optional)

| Variable | Default | Purpose |
|----------|---------|---------|
| `REVIEW_ENGINE` | `claude` | Primary review engine: `claude` or `copilot` |
| `REVIEW_ENGINE` | `claude` | Primary review engine: `claude`, `gemini`, or `copilot` |
| `LIVE_MODE` | `false` | If `true`, reviews are posted live; if `false`, dry-run only |
| `DELEGATION_ORGS` | Empty | Comma-separated orgs where AI can auto-fix: `petry-projects,don-petry` |
| `MAX_REVIEW_CYCLES` | `3` | Max review iterations before escalating to human |
Expand Down
41 changes: 39 additions & 2 deletions scripts/engine.sh
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,18 @@ case "$REVIEW_ENGINE" in
DUCK_ENGINE="copilot"
DUCK_MODEL="gpt-5.4"
;;
gemini)
ENGINE_TRIAGE_MODEL="gemini-2.0-flash"
ENGINE_DEEP_MODEL="gemini-1.5-pro"
ENGINE_AUDIT_MODEL="gemini-1.5-pro"
ENGINE_ACTION_MODEL="gemini-1.5-pro"
ENGINE_SINGLE_MODEL="gemini-1.5-pro"
ENGINE_LABEL="triage: gemini-2.0-flash → deep: gemini-1.5-pro + duck: sonnet 4.6 → audit: gemini-1.5-pro"
ENGINE_SINGLE_LABEL="single-reviewer mode: gemini-1.5-pro"
# Cross-engine rubber duck: use Claude for diversity
DUCK_ENGINE="claude"
DUCK_MODEL="claude-sonnet-4-6"
Comment thread
don-petry marked this conversation as resolved.
;;
copilot)
Comment thread
don-petry marked this conversation as resolved.
ENGINE_TRIAGE_MODEL="gpt-5-mini"
ENGINE_DEEP_MODEL="gpt-5.2"
Expand All @@ -54,7 +66,7 @@ case "$REVIEW_ENGINE" in
DUCK_MODEL="claude-sonnet-4-6"
;;
*)
echo "::error::Unknown REVIEW_ENGINE='$REVIEW_ENGINE' (expected: claude or copilot)"
echo "::error::Unknown REVIEW_ENGINE='$REVIEW_ENGINE' (expected: claude, gemini, or copilot)"
exit 1
;;
esac
Expand All @@ -71,7 +83,7 @@ echo " engine: $REVIEW_ENGINE ($ENGINE_LABEL)"
# review-one-pr.sh exits with code 2 when this fires so the caller can switch engines.
is_rate_limited() {
local text="$1"
echo "$text" | grep -qiE "(hit your limit|rate[ -]?limit|resets [0-9]+(am|pm)|usage limit|quota exceeded|too many requests|exceeded.*quota)"
echo "$text" | grep -qiE "(hit your limit|rate[ -]?limit|resets [0-9]+(am|pm)|usage limit|quota exceeded|too many requests|exceeded.*quota|429|exhausted)"
Comment thread
don-petry marked this conversation as resolved.
Outdated
}

# is_transient_failure <exit_code>
Expand Down Expand Up @@ -112,6 +124,13 @@ run_triage() {
--disallowed-tools "Bash,Read,Write,Edit,Grep,Glob,WebFetch,WebSearch,Task,TodoWrite,NotebookEdit" \
< "$prompt_file" || rc=$?
;;
gemini)
timeout "$TRIAGE_TIMEOUT_SEC" gemini --prompt "" \
--model "$ENGINE_TRIAGE_MODEL" \
--approval-mode plan \
Comment thread
don-petry marked this conversation as resolved.
Outdated
--output-format text \
< "$prompt_file" || rc=$?
;;
copilot)
# gh copilot is now a built-in; auth via GH_PAT (user token with Copilot subscription).
# Model selection is not supported by gh copilot suggest — uses Copilot's default.
Expand Down Expand Up @@ -155,6 +174,13 @@ run_agentic() {
--allowed-tools "Bash,Read,Grep,Glob" \
< "$prompt_file"
;;
gemini)
timeout "$DEEP_TIMEOUT_SEC" gemini --prompt "" \
--model "$model" \
--approval-mode auto_edit \
--output-format text \
< "$prompt_file"
;;
copilot)
# gh copilot is now a built-in; auth via GH_PAT (user token with Copilot subscription).
# Model selection is not supported by gh copilot suggest — uses Copilot's default.
Expand Down Expand Up @@ -209,15 +235,26 @@ run_duck() {
case "$DUCK_ENGINE" in
claude)
unset COPILOT_GITHUB_TOKEN 2>/dev/null || true
unset GOOGLE_API_KEY 2>/dev/null || true
timeout "$DUCK_TIMEOUT_SEC" claude --print \
--model "$model" \
--permission-mode acceptEdits \
--allowed-tools "Bash,Read,Grep,Glob" \
--max-turns 25 \
< "$prompt_file"
;;
gemini)
unset CLAUDE_CODE_OAUTH_TOKEN 2>/dev/null || true
unset COPILOT_GITHUB_TOKEN 2>/dev/null || true
timeout "$DUCK_TIMEOUT_SEC" gemini --prompt "" \
--model "$model" \
--approval-mode auto_edit \
--output-format text \
< "$prompt_file"
;;
copilot)
unset CLAUDE_CODE_OAUTH_TOKEN 2>/dev/null || true
unset GOOGLE_API_KEY 2>/dev/null || true
# gh copilot is now a built-in; auth via GH_PAT (user token with Copilot subscription).
( export GH_TOKEN="$COPILOT_GITHUB_TOKEN"
timeout "$DUCK_TIMEOUT_SEC" gh copilot suggest "$(cat "$prompt_file")" --agent shell
Expand Down
18 changes: 13 additions & 5 deletions scripts/review-batch.sh
Original file line number Diff line number Diff line change
Expand Up @@ -53,16 +53,24 @@ while IFS= read -r pr_url; do
rc=0
bash scripts/review-one-pr.sh "$pr_url" || rc=$?

# Exit code 2 = engine rate-limited. Switch from claude to copilot once,
# then retry this PR. All later PRs in the batch use copilot too.
# Exit code 2 = engine rate-limited.
# Fallback chain: claude -> gemini -> copilot
if [ "$rc" -eq 2 ] && [ "${REVIEW_ENGINE:-claude}" = "claude" ]; then
if ! gh extension list 2>/dev/null | grep -q copilot; then
echo "::warning::Copilot fallback engine unavailable (not installed) — skipping $pr_url and continuing batch"
echo "::warning::Claude rate limit hit — switching to Gemini engine for remaining PRs"
export REVIEW_ENGINE=gemini
engine_fallbacks=$((engine_fallbacks + 1))
rc=0
bash scripts/review-one-pr.sh "$pr_url" || rc=$?
Comment thread
don-petry marked this conversation as resolved.
Outdated
fi
Comment thread
don-petry marked this conversation as resolved.

if [ "$rc" -eq 2 ] && [ "${REVIEW_ENGINE}" = "gemini" ]; then
if ! gh extension list 2>/dev/null | grep -q copilot && ! gh copilot --version > /dev/null 2>&1; then
echo "::warning::Copilot fallback engine unavailable — skipping $pr_url and continuing batch"
failed=$((failed + 1))
echo "::endgroup::"
continue
fi
echo "::warning::Claude rate limit hit — switching to Copilot engine for remaining PRs"
echo "::warning::Gemini rate limit hit — switching to Copilot engine for remaining PRs"
export REVIEW_ENGINE=copilot
engine_fallbacks=$((engine_fallbacks + 1))
rc=0
Expand Down
3 changes: 2 additions & 1 deletion scripts/review-one-pr.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,9 @@
#
# Env:
# GH_TOKEN — set by the workflow
# REVIEW_ENGINE — "claude" or "copilot" (default: claude)
# REVIEW_ENGINE — "claude", "gemini", or "copilot" (default: claude)
# CLAUDE_CODE_OAUTH_TOKEN — (claude engine) set by the workflow
# GOOGLE_API_KEY — (gemini engine) set by the workflow
# COPILOT_GITHUB_TOKEN — (copilot engine) set by the workflow
# DRY_RUN — "true" or "false"
#
Expand Down
Loading