Skip to content
Merged
14 changes: 13 additions & 1 deletion .github/workflows/pr-review.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,8 @@ jobs:
CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
# Copilot engine reuses GH_PAT (a separate user PAT with Copilot subscription).
COPILOT_GITHUB_TOKEN: ${{ secrets.GH_PAT }}
# Gemini engine auth — requires GOOGLE_API_KEY repo secret.
GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
# Default to dry-run. workflow_dispatch can override; repository_dispatch
# (mention) always runs live; vars.LIVE_MODE=true switches scheduled runs live.
DRY_RUN: ${{ github.event_name == 'repository_dispatch' && 'false' || inputs.dry_run || (vars.LIVE_MODE == 'true' && 'false' || 'true') }}
Expand Down Expand Up @@ -107,13 +109,23 @@ jobs:
case "$REVIEW_ENGINE" in
claude)
install_claude
# Best-effort install of fallbacks for duck/fallback support
npm install -g @google/gemini-cli || true
if ! env GH_TOKEN="$COPILOT_GITHUB_TOKEN" gh copilot --version > /dev/null 2>&1; then
echo "::warning::gh copilot built-in unavailable with user token — rate-limit fallback will be unavailable."
fi
Comment thread
don-petry marked this conversation as resolved.
Resolved
;;
gemini)
npm install -g @google/gemini-cli
install_claude || true
;;
copilot)
gh extension install github/gh-copilot
# Best-effort install of copilot if built-in is missing
if ! gh copilot --version > /dev/null 2>&1; then
gh extension install github/gh-copilot || true
fi
install_claude || true
npm install -g @google/gemini-cli || true
;;
*)
echo "::error::Unknown REVIEW_ENGINE=$REVIEW_ENGINE"
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,6 @@ Mention trigger: Comment `@petry-review-bot` on any org PR for immediate review.

## Documentation

- [Architecture & Capabilities](AGENT.md)
- [Setup Guide](SETUP.md)
- [Machine User Setup](MACHINE_USER_SETUP.md)
- [Architecture & Capabilities](docs/pr-review-agent/pr-review-agent.md)
- [Setup Guide](docs/pr-review-agent/setup.md)
- [Machine User Setup](docs/pr-review-agent/machine-user-setup.md)
File renamed without changes.
76 changes: 38 additions & 38 deletions DOCUMENTATION.md → docs/pr-review-agent/documentation-index.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,20 +3,20 @@
This repository contains several documentation files describing the PR Review Agent system. Start here to understand which document you need.

## Quick Start
- **[SETUP.md](SETUP.md)** ⭐ Start here!
- **[setup.md](setup.md)** ⭐ Start here!
- Quick reference for configuration and usage
- Lists required secrets and variables
- Shows how to run manually or check logs
- Includes troubleshooting for common issues

## Understanding the System
- **[AGENT.md](AGENT.md)** — Full system documentation
- **[pr-review-agent.md](pr-review-agent.md)** — Full system documentation
- Architecture and design philosophy
- How PR reviews are performed
- Agent capabilities and limitations
- Configuration options

- **[IMPLEMENTATION.md](IMPLEMENTATION.md)** — Technical deep dive
- **[implementation.md](implementation.md)** — Technical deep dive
- Authentication: machine user with PAT
- How PR enumeration works
- Review pipeline architecture
Expand All @@ -25,7 +25,7 @@ This repository contains several documentation files describing the PR Review Ag
- Stuck PR cleanup explained

## Setting Up Authentication
- **[MACHINE_USER_SETUP.md](MACHINE_USER_SETUP.md)** — Machine user and PAT setup
- **[machine-user-setup.md](machine-user-setup.md)** — Machine user and PAT setup
- Create machine user account and org team
- Configure CODEOWNERS for code owner approvals
- Generate fine-grained PAT
Expand All @@ -35,49 +35,49 @@ This repository contains several documentation files describing the PR Review Ag

```
.github/workflows/
├── pr-review.yml # Main hourly review workflow
└── fix-stuck-prs.yml # Cleanup for stuck PRs
├── pr-review.yml # Main hourly review workflow
└── fix-stuck-prs.yml # Cleanup for stuck PRs

scripts/
├── list-prs.sh # Find candidate PRs
├── review-one-pr.sh # Orchestrate review
├── post-pr-review.sh # Post approval and auto-merge
└── fix-stuck-prs.sh # Fix old stuck PRs
├── list-prs.sh # Find candidate PRs
├── review-one-pr.sh # Orchestrate review
├── post-pr-review.sh # Post approval and auto-merge
└── fix-stuck-prs.sh # Fix old stuck PRs

prompts/
├── shared.md # Shared context
├── triage.md # Quick assessment
├── deep-review.md # Code analysis
├── security-audit.md # Vulnerability check
└── synthesize.md # Final verdict

docs/
├── SETUP.md # Quick start (read first)
├── AGENT.md # Full documentation
├── IMPLEMENTATION.md # Technical details
├── MACHINE_USER_SETUP.md # Machine user and PAT setup
└── DOCUMENTATION.md # This file
├── shared.md # Shared context
├── triage.md # Quick assessment
├── deep-review.md # Code analysis
├── security-audit.md # Vulnerability check
└── synthesize.md # Final verdict

docs/pr-review-agent/
├── setup.md # Quick start (read first)
├── pr-review-agent.md # Full documentation
├── implementation.md # Technical details
├── machine-user-setup.md # Machine user and PAT setup
└── documentation-index.md # This file
```

## Common Tasks

### I want to understand what this agent does
→ Read [AGENT.md](AGENT.md)
→ Read [pr-review-agent.md](pr-review-agent.md)

### I need to set up the agent in a new organization
→ Follow [MACHINE_USER_SETUP.md](MACHINE_USER_SETUP.md)
→ Follow [machine-user-setup.md](machine-user-setup.md)

### The agent isn't working, help!
→ Check [SETUP.md#troubleshooting](SETUP.md#troubleshooting)
→ Check [setup.md#troubleshooting](setup.md#troubleshooting)

### I want to understand the architecture
→ Read [IMPLEMENTATION.md](IMPLEMENTATION.md)
→ Read [implementation.md](implementation.md)

### I need to update configuration
→ See [SETUP.md#repository-variables](SETUP.md#repository-variables)
→ See [setup.md#repository-variables](setup.md#repository-variables)

### I want to run a manual review
→ See [SETUP.md#running-manually](SETUP.md#running-manually)
→ See [setup.md#running-manually](setup.md#running-manually)

## Authentication Method

Expand All @@ -97,9 +97,9 @@ All secrets must be set in the repository (`Settings → Secrets and variables

| Secret | Purpose |
|--------|---------|
| `CLAUDE_CODE_OAUTH_TOKEN` | Claude Code API access |
| `DON_PETRY_BOT_PETRY_PROJECT_PAT` | Machine user fine-grained PAT |
| `COPILOT_GITHUB_TOKEN` | Optional fallback engine |
| `CLAUDE_CODE_OAUTH_TOKEN` | Claude Code authentication |
| `DON_PETRY_BOT_GH_PAT` | Machine user fine-grained PAT |
| `GH_PAT` | User PAT with Copilot subscription |

## Review Workflow

Expand All @@ -121,17 +121,17 @@ All secrets must be set in the repository (`Settings → Secrets and variables

## Support and Troubleshooting

- **Workflow failing to authenticate**: Check [SETUP.md#troubleshooting](SETUP.md#troubleshooting)
- **Questions about design**: See [IMPLEMENTATION.md](IMPLEMENTATION.md)
- **Setup instructions**: Follow [MACHINE_USER_SETUP.md](MACHINE_USER_SETUP.md)
- **Agent capabilities**: Read [AGENT.md](AGENT.md)
- **Workflow failing to authenticate**: Check [setup.md#troubleshooting](setup.md#troubleshooting)
- **Questions about design**: See [implementation.md](implementation.md)
- **Setup instructions**: Follow [machine-user-setup.md](machine-user-setup.md)
- **Agent capabilities**: Read [pr-review-agent.md](pr-review-agent.md)

## Document Maintenance

These documents are kept in sync with the actual implementation. When updating the system:
1. Update the relevant scripts/workflows
2. Update the corresponding documentation
3. Ensure SETUP.md reflects current state
4. Update IMPLEMENTATION.md if architecture changes
3. Ensure setup.md reflects current state
4. Update implementation.md if architecture changes

Last updated: April 26, 2026
Last updated: May 11, 2026
34 changes: 20 additions & 14 deletions IMPLEMENTATION.md → docs/pr-review-agent/implementation.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,13 @@ This document describes the current implementation of the PR Review Agent and ke

### Token Usage

All workflows use the `DON_PETRY_BOT_PETRY_PROJECT_PAT` org secret directly:
All workflows use the `DON_PETRY_BOT_GH_PAT` org secret directly:
```yaml
env:
GH_TOKEN: ${{ secrets.DON_PETRY_BOT_PETRY_PROJECT_PAT }}
GH_TOKEN: ${{ secrets.DON_PETRY_BOT_GH_PAT }}
```

The PAT is a fine-grained token scoped to the `petry-projects` organization with 90-day expiry. See [MACHINE_USER_SETUP.md](MACHINE_USER_SETUP.md) for rotation instructions.
The PAT is a fine-grained token scoped to the `petry-projects` organization with 90-day expiry. See [MACHINE_USER_SETUP.md](machine-user-setup.md) for rotation instructions.

## PR Enumeration

Expand Down Expand Up @@ -162,25 +162,30 @@ This script runs with machine user PAT credentials that can approve PRs across t

## Rate Limiting and Fallback

The `pr-review.yml` workflow includes fallback logic:
The `scripts/review-batch.sh` script includes fallback logic:

```bash
# Fallback chain: claude -> gemini -> copilot
if [ "$rc" -eq 2 ] && [ "${REVIEW_ENGINE:-claude}" = "claude" ]; then
echo "Claude rate limit hit — switching to Copilot engine for remaining PRs"
export REVIEW_ENGINE=gemini
# ... retry this PR with gemini ...
fi

if [ "$rc" -eq 2 ] && [ "${REVIEW_ENGINE}" = "gemini" ]; then
export REVIEW_ENGINE=copilot
bash scripts/review-one-pr.sh "$pr_url" || rc=$?
# ... retry this PR with copilot ...
fi
```

If Claude hits rate limits, the workflow switches to GitHub Copilot for remaining PRs in the batch. This ensures reviews continue even under high load.
If Claude hits rate limits, the batch switches to Gemini, and then to GitHub Copilot if needed. This ensures reviews continue even under high load.

## Metrics and Monitoring

Key metrics tracked in workflow logs:
- **Reviews posted:** Number of PRs actually approved
- **No-ops skipped:** PRs already reviewed, not re-reviewed
- **Failures:** PRs that had errors during review
- **Engine fallbacks:** Times Claude rate limit triggered Copilot fallback
- **Engine fallbacks:** Cumulative count and specific engines used (e.g., \"gemini, copilot\")
Comment thread
don-petry marked this conversation as resolved.
Resolved

View recent runs:
```bash
Expand All @@ -198,7 +203,7 @@ gh run view <run-id> --repo don-petry/pr-review-agent --log

| Variable | Default | Purpose |
|----------|---------|---------|
| `REVIEW_ENGINE` | `claude` | Primary review engine |
| `REVIEW_ENGINE` | `claude` | Primary review engine: `claude`, `gemini`, or `copilot` |
| `DRY_RUN` | `true` | If true, don't post reviews |
| `MAX_PRS` | `10` | Max reviews per run |
| `CANDIDATE_LIMIT` | `100` | Max candidates scanned |
Expand All @@ -209,9 +214,10 @@ gh run view <run-id> --repo don-petry/pr-review-agent --log

| Secret | Purpose |
|--------|---------|
| `DON_PETRY_BOT_GH_PAT` | Machine user PAT for GitHub API access (BOT_USER) |
| `GH_PAT` | User PAT with Copilot subscription (fallback source) |
| `CLAUDE_CODE_OAUTH_TOKEN` | Claude Code authentication |
| `DON_PETRY_BOT_PETRY_PROJECT_PAT` | Machine user PAT for GitHub API access |
| `COPILOT_GITHUB_TOKEN` | GitHub Copilot fallback token |
| `GOOGLE_API_KEY` | Gemini API authentication |

## File Structure

Expand All @@ -234,9 +240,9 @@ gh run view <run-id> --repo don-petry/pr-review-agent --log
│ ├── synthesize.md # Combine findings, make decision
│ └── cascade-action.md # Review coordination (legacy)
│
├── SETUP.md # Quick start guide
├── MACHINE_USER_SETUP.md # Machine user and PAT setup
├── IMPLEMENTATION.md # This file
├── setup.md # Quick start guide
├── machine-user-setup.md # Machine user and PAT setup
├── implementation.md # This file
└── README.md # Overview
```

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,13 +45,13 @@ Or use path-specific rules as needed.
## Step 4: Store the PAT as an Org Secret

```bash
gh secret set DON_PETRY_BOT_PETRY_PROJECT_PAT --org petry-projects --body "<paste-token>"
gh secret set DON_PETRY_BOT_GH_PAT --org petry-projects --body "<paste-token>"
Comment thread
don-petry marked this conversation as resolved.
Resolved
```

Or store at repo level if preferred:

```bash
gh secret set DON_PETRY_BOT_PETRY_PROJECT_PAT --repo don-petry/pr-review-agent --body "<paste-token>"
gh secret set DON_PETRY_BOT_GH_PAT --repo don-petry/pr-review-agent --body "<paste-token>"
```

### Verify the secret is set:
Expand All @@ -60,7 +60,7 @@ gh secret set DON_PETRY_BOT_PETRY_PROJECT_PAT --repo don-petry/pr-review-agent -
gh secret list --repo don-petry/pr-review-agent
```

Should show `DON_PETRY_BOT_PETRY_PROJECT_PAT` in the list.
Should show `DON_PETRY_BOT_GH_PAT` in the list.

## Step 5: Test the Setup

Expand Down Expand Up @@ -92,7 +92,7 @@ Fine-grained PATs have a configurable expiry (90-day recommended). To rotate:

1. Sign in as the machine user
2. Generate a new fine-grained PAT with the same scopes
3. Update the secret: `gh secret set DON_PETRY_BOT_PETRY_PROJECT_PAT --repo don-petry/pr-review-agent --body "<new-token>"`
3. Update the secret: `gh secret set DON_PETRY_BOT_GH_PAT --repo don-petry/pr-review-agent --body "<new-token>"`
4. Revoke the old token

## Trade-offs vs GitHub App
Expand All @@ -110,15 +110,15 @@ Fine-grained PATs have a configurable expiry (90-day recommended). To rotate:
### Issue: "Resource not accessible by integration"
- Verify the PAT has the required scopes
- Check the machine user has access to the target repos
- Ensure `DON_PETRY_BOT_PETRY_PROJECT_PAT` secret is set correctly
- Ensure `DON_PETRY_BOT_GH_PAT` secret is set correctly

### Issue: Approval doesn't satisfy code owner requirement
- Verify the machine user is in the team listed in CODEOWNERS
- Check the repo's CODEOWNERS file is valid (`gh api repos/OWNER/REPO/codeowners/errors`)
- Ensure branch protection has `require_code_owner_review: true`

### Issue: Token expired
- Generate a new fine-grained PAT and update the `DON_PETRY_BOT_PETRY_PROJECT_PAT` secret
- Generate a new fine-grained PAT and update the `DON_PETRY_BOT_GH_PAT` secret

## References

Expand Down
Loading