Skip to content
Merged
14 changes: 13 additions & 1 deletion .github/workflows/pr-review.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,8 @@ jobs:
CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
# Copilot engine reuses GH_PAT (a separate user PAT with Copilot subscription).
COPILOT_GITHUB_TOKEN: ${{ secrets.GH_PAT }}
# Gemini engine auth — requires GOOGLE_API_KEY repo secret.
GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
# Default to dry-run. workflow_dispatch can override; repository_dispatch
# (mention) always runs live; vars.LIVE_MODE=true switches scheduled runs live.
DRY_RUN: ${{ github.event_name == 'repository_dispatch' && 'false' || inputs.dry_run || (vars.LIVE_MODE == 'true' && 'false' || 'true') }}
Expand Down Expand Up @@ -107,13 +109,23 @@ jobs:
case "$REVIEW_ENGINE" in
claude)
install_claude
# Best-effort install of fallbacks for duck/fallback support
npm install -g @google/gemini-cli || true
if ! env GH_TOKEN="$COPILOT_GITHUB_TOKEN" gh copilot --version > /dev/null 2>&1; then
echo "::warning::gh copilot built-in unavailable with user token — rate-limit fallback will be unavailable."
fi
Comment thread
don-petry marked this conversation as resolved.
;;
gemini)
npm install -g @google/gemini-cli
install_claude || true
;;
copilot)
gh extension install github/gh-copilot
# Best-effort install of copilot if built-in is missing
if ! gh copilot --version > /dev/null 2>&1; then
gh extension install github/gh-copilot || true
fi
install_claude || true
npm install -g @google/gemini-cli || true
;;
*)
echo "::error::Unknown REVIEW_ENGINE=$REVIEW_ENGINE"
Expand Down
46 changes: 31 additions & 15 deletions AGENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ A scheduled GitHub Action that reviews open PRs on don-petry's behalf.
Runs hourly, classifies risk, auto-approves low/medium-risk PRs that pass all
quality gates, and escalates high-risk or gated PRs for human review.

Supports two LLM engines via the `REVIEW_ENGINE` repo variable: **Claude** (default)
Supports three LLM engines via the `REVIEW_ENGINE` repo variable: **Claude** (default), **Gemini**,
and **Copilot**.

## How it works
Expand Down Expand Up @@ -46,23 +46,27 @@ and **Copilot**.
model tiers. Different model families have different blind spots — running
both catches issues that either alone would miss.

The rubber duck is **always the opposite engine**: if `REVIEW_ENGINE=claude`,
the duck is Copilot (GPT-5.4), and vice versa. No extra configuration needed.
The rubber duck is **always a diverse engine**:
- If `REVIEW_ENGINE=claude`, the duck is Copilot (GPT-5.4).
- If `REVIEW_ENGINE=gemini`, the duck is Claude (Sonnet 4.6).
- If `REVIEW_ENGINE=copilot`, the duck is Claude (Sonnet 4.6).

No extra configuration needed.

**Graceful degradation:** if the rubber duck fails (missing credentials,
CLI not installed, timeout), the cascade continues with the primary deep
review only. The duck is additive, never blocking.

### Engine model mapping

| Tier | Claude primary | Copilot primary |
|---|---|---|
| Triage | Haiku 4.5 | GPT-5-mini |
| Deep review | Sonnet 4.6 | GPT-5.2 |
| Rubber duck | GPT-5.4 (cross) | Sonnet 4.6 (cross) |
| Synthesis | Sonnet 4.6 | GPT-5.2 |
| Security audit | Opus 4.6 | GPT-5.4 |
| Action / single review | Sonnet 4.6 / Opus 4.6 | GPT-5.2 / GPT-5.4 |
| Tier | Claude primary | Gemini primary | Copilot primary |
|---|---|---|---|
| Triage | Haiku 4.5 | Gemini 2.0 Flash | GPT-5-mini |
| Deep review | Sonnet 4.6 | Gemini 1.5 Pro | GPT-5.2 |
| Rubber duck | GPT-5.4 (cross) | Sonnet 4.6 (cross) | Sonnet 4.6 (cross) |
| Synthesis | Sonnet 4.6 | Gemini 1.5 Pro | GPT-5.2 |
| Security audit | Opus 4.6 | Gemini 1.5 Pro | GPT-5.4 |
| Action / single review | Sonnet 4.6 / Opus 4.6 | Gemini 1.5 Pro | GPT-5.2 / GPT-5.4 |

**Cost profile:**
- ~80% of PRs: triage + single confirm (2 calls, ~30s)
Expand Down Expand Up @@ -168,6 +172,14 @@ Store as a repo secret:
gh secret set CLAUDE_CODE_OAUTH_TOKEN --repo don-petry/pr-review-agent
```

#### Gemini engine

Requires a Google API Key. Store as a repo secret:

```
gh secret set GOOGLE_API_KEY --repo don-petry/pr-review-agent
```

#### Copilot engine

Create a GitHub PAT with Copilot scope. Store as a repo secret:
Expand All @@ -179,6 +191,9 @@ gh secret set COPILOT_GITHUB_TOKEN --repo don-petry/pr-review-agent
### 4. Choose your engine

```
# Use Gemini:
gh variable set REVIEW_ENGINE --body gemini --repo don-petry/pr-review-agent

# Use Copilot (GPT models):
gh variable set REVIEW_ENGINE --body copilot --repo don-petry/pr-review-agent

Expand Down Expand Up @@ -225,7 +240,7 @@ gh workflow run pr-review.yml --repo don-petry/pr-review-agent -f dry_run=false

## Tuning

- **Review engine** — `REVIEW_ENGINE` repo variable: `claude` (default) or
- **Review engine** — `REVIEW_ENGINE` repo variable: `claude` (default), `gemini`, or
`copilot`. Controls which CLI and model family is used.
- **Risk rules** — edit `prompts/shared.md` (taxonomy), or the per-tier
prompts (`prompts/deep-review.md`, `prompts/security-audit.md`).
Expand Down Expand Up @@ -287,7 +302,7 @@ PR comment "@petry-review-bot please review"
## Architecture

```
scripts/engine.sh ← LLM abstraction (claude/copilot dispatch)
scripts/engine.sh ← LLM abstraction (claude/gemini/copilot dispatch)
scripts/review-one-pr.sh ← Cascade orchestrator (sources engine.sh)
scripts/list-prs.sh ← PR enumeration

Expand All @@ -302,8 +317,9 @@ prompts/shared.md ← Shared risk taxonomy and decision gates
## Cost

Uses the configured engine's billing:
- **Claude:** Max plan via OAuth token — no per-token API billing.
- **Copilot:** Included in GitHub Copilot subscription.
- **Claude**: Max plan via OAuth token — no per-token API billing.
- **Gemini**: API-based billing via `GOOGLE_API_KEY`.
- **Copilot**: Included in GitHub Copilot subscription.

GitHub Actions cost is ~720 runs/month (hourly × 30 days). Runs with zero
candidate PRs finish in ~10s. Each PR reviewed costs ~2-5 min of runner time
Expand Down
22 changes: 14 additions & 8 deletions IMPLEMENTATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,25 +162,30 @@ This script runs with machine user PAT credentials that can approve PRs across t

## Rate Limiting and Fallback

The `pr-review.yml` workflow includes fallback logic:
The `scripts/review-batch.sh` script includes fallback logic:

```bash
# Fallback chain: claude -> gemini -> copilot
if [ "$rc" -eq 2 ] && [ "${REVIEW_ENGINE:-claude}" = "claude" ]; then
echo "Claude rate limit hit — switching to Copilot engine for remaining PRs"
export REVIEW_ENGINE=gemini
# ... retry this PR with gemini ...
fi

if [ "$rc" -eq 2 ] && [ "${REVIEW_ENGINE}" = "gemini" ]; then
export REVIEW_ENGINE=copilot
bash scripts/review-one-pr.sh "$pr_url" || rc=$?
# ... retry this PR with copilot ...
fi
```

If Claude hits rate limits, the workflow switches to GitHub Copilot for remaining PRs in the batch. This ensures reviews continue even under high load.
If Claude hits rate limits, the batch switches to Gemini, and then to GitHub Copilot if needed. This ensures reviews continue even under high load.

## Metrics and Monitoring

Key metrics tracked in workflow logs:
- **Reviews posted:** Number of PRs actually approved
- **No-ops skipped:** PRs already reviewed, not re-reviewed
- **Failures:** PRs that had errors during review
- **Engine fallbacks:** Times Claude rate limit triggered Copilot fallback
- **Engine fallbacks:** Cumulative count and specific engines used (e.g., \"gemini, copilot\")

View recent runs:
```bash
Expand All @@ -198,7 +203,7 @@ gh run view <run-id> --repo don-petry/pr-review-agent --log

| Variable | Default | Purpose |
|----------|---------|---------|
| `REVIEW_ENGINE` | `claude` | Primary review engine |
| `REVIEW_ENGINE` | `claude` | Primary review engine: `claude`, `gemini`, or `copilot` |
| `DRY_RUN` | `true` | If true, don't post reviews |
| `MAX_PRS` | `10` | Max reviews per run |
| `CANDIDATE_LIMIT` | `100` | Max candidates scanned |
Expand All @@ -209,9 +214,10 @@ gh run view <run-id> --repo don-petry/pr-review-agent --log

| Secret | Purpose |
|--------|---------|
| `DON_PETRY_BOT_GH_PAT` | Machine user PAT for GitHub API access (BOT_USER) |
| `GH_PAT` | User PAT with Copilot subscription (fallback source) |
| `CLAUDE_CODE_OAUTH_TOKEN` | Claude Code authentication |
| `DON_PETRY_BOT_PETRY_PROJECT_PAT` | Machine user PAT for GitHub API access |
| `COPILOT_GITHUB_TOKEN` | GitHub Copilot fallback token |
| `GOOGLE_API_KEY` | Gemini API authentication |

## File Structure

Expand Down
12 changes: 7 additions & 5 deletions SETUP.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
# PR Review Agent Setup

This repository automates PR reviews for the `petry-projects` organization using Claude Code or GitHub Copilot.
This repository automates PR reviews for the `petry-projects` organization using
Claude Code, Google Gemini, or GitHub Copilot.

## Quick Start

### Prerequisites
- GitHub organization: `petry-projects`
- Machine user account (e.g., `petry-pr-bot`) added to org team in CODEOWNERS
- Machine user account (e.g., `don-petry-bot`) added to org team in CODEOWNERS
- Secrets configured in the repository

### Repository Secrets Required
Expand All @@ -15,15 +16,16 @@ Store these in the repository settings (`Settings → Secrets and variables →

| Secret | Description | Source |
|--------|-------------|--------|
| `DON_PETRY_BOT_GH_PAT` | Machine user PAT for GitHub API access | Generated from machine user account settings |
| \`GH_PAT\` | User PAT with Copilot subscription (for fallback) | GitHub PAT with Copilot scope |
| `CLAUDE_CODE_OAUTH_TOKEN` | Claude Code authentication token | `claude setup-token` |
| `DON_PETRY_BOT_PETRY_PROJECT_PAT` | Machine user fine-grained PAT | Generated from machine user account settings |
| `COPILOT_GITHUB_TOKEN` | GitHub Copilot token (optional, for fallback) | GitHub PAT with Copilot scope |
| `GOOGLE_API_KEY` | Gemini API authentication token | Google AI Studio / Vertex AI |

### Repository Variables (Optional)

| Variable | Default | Purpose |
|----------|---------|---------|
| `REVIEW_ENGINE` | `claude` | Primary review engine: `claude` or `copilot` |
| `REVIEW_ENGINE` | `claude` | Primary review engine: `claude`, `gemini`, or `copilot` |
| `LIVE_MODE` | `false` | If `true`, reviews are posted live; if `false`, dry-run only |
| `DELEGATION_ORGS` | Empty | Comma-separated orgs where AI can auto-fix: `petry-projects,don-petry` |
| `MAX_REVIEW_CYCLES` | `3` | Max review iterations before escalating to human |
Expand Down
45 changes: 41 additions & 4 deletions scripts/engine.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
# Engine abstraction layer for LLM invocations.
# Supports: claude, copilot
# Supports: claude, gemini, copilot
#
# Sourced by review-one-pr.sh — provides:
# run_triage <prompt_file> — no-tool tier (stdout capture)
Expand Down Expand Up @@ -41,6 +41,18 @@ case "$REVIEW_ENGINE" in
DUCK_ENGINE="copilot"
DUCK_MODEL="gpt-5.4"
;;
gemini)
ENGINE_TRIAGE_MODEL="gemini-2.0-flash"
ENGINE_DEEP_MODEL="gemini-1.5-pro"
ENGINE_AUDIT_MODEL="gemini-1.5-pro"
ENGINE_ACTION_MODEL="gemini-1.5-pro"
ENGINE_SINGLE_MODEL="gemini-1.5-pro"
ENGINE_LABEL="triage: gemini-2.0-flash → deep: gemini-1.5-pro + duck: sonnet 4.6 → audit: gemini-1.5-pro"
ENGINE_SINGLE_LABEL="single-reviewer mode: gemini-1.5-pro"
# Cross-engine rubber duck: use Claude for diversity
DUCK_ENGINE="claude"
DUCK_MODEL="claude-sonnet-4-6"
Comment thread
don-petry marked this conversation as resolved.
;;
copilot)
Comment thread
don-petry marked this conversation as resolved.
ENGINE_TRIAGE_MODEL="gpt-5-mini"
ENGINE_DEEP_MODEL="gpt-5.2"
Expand All @@ -54,7 +66,7 @@ case "$REVIEW_ENGINE" in
DUCK_MODEL="claude-sonnet-4-6"
;;
*)
echo "::error::Unknown REVIEW_ENGINE='$REVIEW_ENGINE' (expected: claude or copilot)"
echo "::error::Unknown REVIEW_ENGINE='$REVIEW_ENGINE' (expected: claude, gemini, or copilot)"
exit 1
;;
esac
Expand All @@ -67,11 +79,11 @@ export DUCK_ENGINE DUCK_MODEL
echo " engine: $REVIEW_ENGINE ($ENGINE_LABEL)"

# is_rate_limited <text>
# Returns 0 (true) if the text looks like a Claude or Copilot rate-limit message.
# Returns 0 (true) if the text looks like a Claude, Gemini, or Copilot rate-limit message.
# review-one-pr.sh exits with code 2 when this fires so the caller can switch engines.
is_rate_limited() {
local text="$1"
echo "$text" | grep -qiE "(hit your limit|rate[ -]?limit|resets [0-9]+(am|pm)|usage limit|quota exceeded|too many requests|exceeded.*quota)"
echo "$text" | grep -qiE "(hit your limit|rate[ -]?limit|resets [0-9]+(am|pm)|usage limit|quota exceeded|too many requests|exceeded.*quota|\b429\b|exhausted)"
Comment thread
don-petry marked this conversation as resolved.
Outdated
}

# is_transient_failure <exit_code>
Expand Down Expand Up @@ -112,6 +124,13 @@ run_triage() {
--disallowed-tools "Bash,Read,Write,Edit,Grep,Glob,WebFetch,WebSearch,Task,TodoWrite,NotebookEdit" \
< "$prompt_file" || rc=$?
;;
gemini)
timeout "$TRIAGE_TIMEOUT_SEC" gemini --prompt "" \
--model "$ENGINE_TRIAGE_MODEL" \
--approval-mode plan \
Comment thread
don-petry marked this conversation as resolved.
Outdated
--output-format text \
< "$prompt_file" || rc=$?
;;
copilot)
# gh copilot is now a built-in; auth via GH_PAT (user token with Copilot subscription).
# Model selection is not supported by gh copilot suggest — uses Copilot's default.
Expand Down Expand Up @@ -155,6 +174,13 @@ run_agentic() {
--allowed-tools "Bash,Read,Grep,Glob" \
< "$prompt_file"
;;
gemini)
timeout "$DEEP_TIMEOUT_SEC" gemini --prompt "" \
--model "$model" \
--approval-mode auto_edit \
--output-format text \
< "$prompt_file"
;;
copilot)
# gh copilot is now a built-in; auth via GH_PAT (user token with Copilot subscription).
# Model selection is not supported by gh copilot suggest — uses Copilot's default.
Expand Down Expand Up @@ -209,15 +235,26 @@ run_duck() {
case "$DUCK_ENGINE" in
claude)
unset COPILOT_GITHUB_TOKEN 2>/dev/null || true
unset GOOGLE_API_KEY 2>/dev/null || true
timeout "$DUCK_TIMEOUT_SEC" claude --print \
--model "$model" \
--permission-mode acceptEdits \
--allowed-tools "Bash,Read,Grep,Glob" \
--max-turns 25 \
< "$prompt_file"
;;
gemini)
unset CLAUDE_CODE_OAUTH_TOKEN 2>/dev/null || true
unset COPILOT_GITHUB_TOKEN 2>/dev/null || true
timeout "$DUCK_TIMEOUT_SEC" gemini --prompt "" \
--model "$model" \
--approval-mode auto_edit \
--output-format text \
< "$prompt_file"
;;
copilot)
unset CLAUDE_CODE_OAUTH_TOKEN 2>/dev/null || true
unset GOOGLE_API_KEY 2>/dev/null || true
# gh copilot is now a built-in; auth via GH_PAT (user token with Copilot subscription).
( export GH_TOKEN="$COPILOT_GITHUB_TOKEN"
timeout "$DUCK_TIMEOUT_SEC" gh copilot suggest "$(cat "$prompt_file")" --agent shell
Expand Down
23 changes: 17 additions & 6 deletions scripts/review-batch.sh
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ actual=0
skipped_noops=0
failed=0
engine_fallbacks=0
fallback_engines=""
processed=0
session_aborted=0
abort_pr=""
Expand All @@ -53,18 +54,28 @@ while IFS= read -r pr_url; do
rc=0
bash scripts/review-one-pr.sh "$pr_url" || rc=$?

# Exit code 2 = engine rate-limited. Switch from claude to copilot once,
# then retry this PR. All later PRs in the batch use copilot too.
# Exit code 2 = engine rate-limited.
# Fallback chain: claude -> gemini -> copilot
if [ "$rc" -eq 2 ] && [ "${REVIEW_ENGINE:-claude}" = "claude" ]; then
if ! gh extension list 2>/dev/null | grep -q copilot; then
echo "::warning::Copilot fallback engine unavailable (not installed) — skipping $pr_url and continuing batch"
echo "::warning::Claude rate limit hit — switching to Gemini engine for remaining PRs"
export REVIEW_ENGINE=gemini
engine_fallbacks=$((engine_fallbacks + 1))
fallback_engines="${fallback_engines:+$fallback_engines, }gemini"
rc=0
bash scripts/review-one-pr.sh "$pr_url" || rc=$?
fi
Comment thread
don-petry marked this conversation as resolved.

if [ "$rc" -eq 2 ] && [ "${REVIEW_ENGINE}" = "gemini" ]; then
if ! gh extension list 2>/dev/null | grep -q copilot && ! gh copilot --version > /dev/null 2>&1; then
echo "::warning::Copilot fallback engine unavailable — skipping $pr_url and continuing batch"
failed=$((failed + 1))
echo "::endgroup::"
continue
fi
echo "::warning::Claude rate limit hit — switching to Copilot engine for remaining PRs"
echo "::warning::Gemini rate limit hit — switching to Copilot engine for remaining PRs"
export REVIEW_ENGINE=copilot
engine_fallbacks=$((engine_fallbacks + 1))
fallback_engines="${fallback_engines:+$fallback_engines, }copilot"
rc=0
bash scripts/review-one-pr.sh "$pr_url" || rc=$?
fi
Expand Down Expand Up @@ -102,7 +113,7 @@ done < "$PRS_FILE"

remaining=$((total_candidates - processed))
summary="Summary: $actual reviews posted, $skipped_noops no-ops skipped, $failed failures"
[ "$engine_fallbacks" -gt 0 ] && summary="$summary, $engine_fallbacks engine fallback(s) to copilot"
[ "$engine_fallbacks" -gt 0 ] && summary="$summary, $engine_fallbacks engine fallback(s) to $fallback_engines"
summary="$summary (processed $processed/$total_candidates candidates)"

if [ "$session_aborted" -eq 1 ]; then
Expand Down
3 changes: 2 additions & 1 deletion scripts/review-one-pr.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,9 @@
#
# Env:
# GH_TOKEN — set by the workflow
# REVIEW_ENGINE — "claude" or "copilot" (default: claude)
# REVIEW_ENGINE — "claude", "gemini", or "copilot" (default: claude)
# CLAUDE_CODE_OAUTH_TOKEN — (claude engine) set by the workflow
# GOOGLE_API_KEY — (gemini engine) set by the workflow
# COPILOT_GITHUB_TOKEN — (copilot engine) set by the workflow
# DRY_RUN — "true" or "false"
#
Expand Down
Loading