Skip to content
Merged
16 changes: 14 additions & 2 deletions .github/workflows/pr-review.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,14 +50,16 @@
# signs the cascade's escalation comments, and is filtered out of the
# candidate queue (it can't approve its own PRs). Human reviewers are
# routed via CODEOWNERS at escalation time, not pinned via env.
BOT_USER: ${{ vars.BOT_USER || 'don-petry-bot' }}
BOT_USER: ${{ vars.BOT_USER || 'donpetry-bot' }}
TARGET_ORG: ${{ vars.TARGET_ORG || 'petry-projects' }}
AGENT_REPO: ${{ github.repository }}
# Review engine: "claude" or "copilot". Controls which CLI and models are used.
REVIEW_ENGINE: ${{ vars.REVIEW_ENGINE || 'claude' }}
CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
# Copilot engine reuses GH_PAT (a separate user PAT with Copilot subscription).
COPILOT_GITHUB_TOKEN: ${{ secrets.GH_PAT }}
# Gemini engine auth — requires GOOGLE_API_KEY repo secret.
GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
# Default to dry-run. workflow_dispatch can override; repository_dispatch
# (mention) always runs live; vars.LIVE_MODE=true switches scheduled runs live.
DRY_RUN: ${{ github.event_name == 'repository_dispatch' && 'false' || inputs.dry_run || (vars.LIVE_MODE == 'true' && 'false' || 'true') }}
Expand Down Expand Up @@ -107,13 +109,23 @@
case "$REVIEW_ENGINE" in
claude)
install_claude
# Best-effort install of fallbacks for duck/fallback support
npm install -g @google/gemini-cli || true

Check warning on line 113 in .github/workflows/pr-review.yml

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Omitting "--ignore-scripts" can lead to the execution of shell scripts. Make sure it is safe here.

See more on https://sonarcloud.io/project/issues?id=petry-projects_.github-private&issues=AZ4VBrc3IgguPulX-770&open=AZ4VBrc3IgguPulX-770&pullRequest=102

Check warning on line 113 in .github/workflows/pr-review.yml

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Using dependencies without locking resolved versions is security-sensitive.

See more on https://sonarcloud.io/project/issues?id=petry-projects_.github-private&issues=AZ4VBrc3IgguPulX-771&open=AZ4VBrc3IgguPulX-771&pullRequest=102
if ! env GH_TOKEN="$COPILOT_GITHUB_TOKEN" gh copilot --version > /dev/null 2>&1; then
echo "::warning::gh copilot built-in unavailable with user token — rate-limit fallback will be unavailable."
fi
Comment thread
don-petry marked this conversation as resolved.
;;
gemini)
npm install -g @google/gemini-cli

Check warning on line 119 in .github/workflows/pr-review.yml

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Omitting "--ignore-scripts" can lead to the execution of shell scripts. Make sure it is safe here.

See more on https://sonarcloud.io/project/issues?id=petry-projects_.github-private&issues=AZ4VBrc3IgguPulX-772&open=AZ4VBrc3IgguPulX-772&pullRequest=102

Check warning on line 119 in .github/workflows/pr-review.yml

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Using dependencies without locking resolved versions is security-sensitive.

See more on https://sonarcloud.io/project/issues?id=petry-projects_.github-private&issues=AZ4VBrc3IgguPulX-773&open=AZ4VBrc3IgguPulX-773&pullRequest=102
install_claude || true
;;
copilot)
gh extension install github/gh-copilot
# Best-effort install of copilot if built-in is missing
if ! gh copilot --version > /dev/null 2>&1; then
gh extension install github/gh-copilot || true
fi
install_claude || true
npm install -g @google/gemini-cli || true

Check warning on line 128 in .github/workflows/pr-review.yml

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Omitting "--ignore-scripts" can lead to the execution of shell scripts. Make sure it is safe here.

See more on https://sonarcloud.io/project/issues?id=petry-projects_.github-private&issues=AZ4VBrc3IgguPulX-774&open=AZ4VBrc3IgguPulX-774&pullRequest=102

Check warning on line 128 in .github/workflows/pr-review.yml

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Using dependencies without locking resolved versions is security-sensitive.

See more on https://sonarcloud.io/project/issues?id=petry-projects_.github-private&issues=AZ4VBrc3IgguPulX-775&open=AZ4VBrc3IgguPulX-775&pullRequest=102
;;
*)
echo "::error::Unknown REVIEW_ENGINE=$REVIEW_ENGINE"
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,6 @@ Mention trigger: Comment `@petry-review-bot` on any org PR for immediate review.

## Documentation

- [Architecture & Capabilities](AGENT.md)
- [Setup Guide](SETUP.md)
- [Machine User Setup](MACHINE_USER_SETUP.md)
- [Architecture & Capabilities](docs/pr-review-agent/pr-review-agent.md)
- [Setup Guide](docs/pr-review-agent/setup.md)
- [Machine User Setup](docs/pr-review-agent/machine-user-setup.md)
File renamed without changes.
76 changes: 38 additions & 38 deletions DOCUMENTATION.md → docs/pr-review-agent/documentation-index.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,20 +3,20 @@
This repository contains several documentation files describing the PR Review Agent system. Start here to understand which document you need.

## Quick Start
- **[SETUP.md](SETUP.md)** ⭐ Start here!
- **[setup.md](setup.md)** ⭐ Start here!
- Quick reference for configuration and usage
- Lists required secrets and variables
- Shows how to run manually or check logs
- Includes troubleshooting for common issues

## Understanding the System
- **[AGENT.md](AGENT.md)** — Full system documentation
- **[pr-review-agent.md](pr-review-agent.md)** — Full system documentation
- Architecture and design philosophy
- How PR reviews are performed
- Agent capabilities and limitations
- Configuration options

- **[IMPLEMENTATION.md](IMPLEMENTATION.md)** — Technical deep dive
- **[implementation.md](implementation.md)** — Technical deep dive
- Authentication: machine user with PAT
- How PR enumeration works
- Review pipeline architecture
Expand All @@ -25,7 +25,7 @@ This repository contains several documentation files describing the PR Review Ag
- Stuck PR cleanup explained

## Setting Up Authentication
- **[MACHINE_USER_SETUP.md](MACHINE_USER_SETUP.md)** — Machine user and PAT setup
- **[machine-user-setup.md](machine-user-setup.md)** — Machine user and PAT setup
- Create machine user account and org team
- Configure CODEOWNERS for code owner approvals
- Generate fine-grained PAT
Expand All @@ -35,49 +35,49 @@ This repository contains several documentation files describing the PR Review Ag

```
.github/workflows/
├── pr-review.yml # Main hourly review workflow
└── fix-stuck-prs.yml # Cleanup for stuck PRs
├── pr-review.yml # Main hourly review workflow
└── fix-stuck-prs.yml # Cleanup for stuck PRs

scripts/
├── list-prs.sh # Find candidate PRs
├── review-one-pr.sh # Orchestrate review
├── post-pr-review.sh # Post approval and auto-merge
└── fix-stuck-prs.sh # Fix old stuck PRs
├── list-prs.sh # Find candidate PRs
├── review-one-pr.sh # Orchestrate review
├── post-pr-review.sh # Post approval and auto-merge
└── fix-stuck-prs.sh # Fix old stuck PRs

prompts/
├── shared.md # Shared context
├── triage.md # Quick assessment
├── deep-review.md # Code analysis
├── security-audit.md # Vulnerability check
└── synthesize.md # Final verdict

docs/
├── SETUP.md # Quick start (read first)
├── AGENT.md # Full documentation
├── IMPLEMENTATION.md # Technical details
├── MACHINE_USER_SETUP.md # Machine user and PAT setup
└── DOCUMENTATION.md # This file
├── shared.md # Shared context
├── triage.md # Quick assessment
├── deep-review.md # Code analysis
├── security-audit.md # Vulnerability check
└── synthesize.md # Final verdict

docs/pr-review-agent/
├── setup.md # Quick start (read first)
├── pr-review-agent.md # Full documentation
├── implementation.md # Technical details
├── machine-user-setup.md # Machine user and PAT setup
└── documentation-index.md # This file
```

## Common Tasks

### I want to understand what this agent does
→ Read [AGENT.md](AGENT.md)
→ Read [pr-review-agent.md](pr-review-agent.md)

### I need to set up the agent in a new organization
→ Follow [MACHINE_USER_SETUP.md](MACHINE_USER_SETUP.md)
→ Follow [machine-user-setup.md](machine-user-setup.md)

### The agent isn't working, help!
→ Check [SETUP.md#troubleshooting](SETUP.md#troubleshooting)
→ Check [setup.md#troubleshooting](setup.md#troubleshooting)

### I want to understand the architecture
→ Read [IMPLEMENTATION.md](IMPLEMENTATION.md)
→ Read [implementation.md](implementation.md)

### I need to update configuration
→ See [SETUP.md#repository-variables](SETUP.md#repository-variables)
→ See [setup.md#repository-variables](setup.md#repository-variables)

### I want to run a manual review
→ See [SETUP.md#running-manually](SETUP.md#running-manually)
→ See [setup.md#running-manually](setup.md#running-manually)

## Authentication Method

Expand All @@ -97,9 +97,9 @@ All secrets must be set in the repository (`Settings → Secrets and variables

| Secret | Purpose |
|--------|---------|
| `CLAUDE_CODE_OAUTH_TOKEN` | Claude Code API access |
| `DON_PETRY_BOT_PETRY_PROJECT_PAT` | Machine user fine-grained PAT |
| `COPILOT_GITHUB_TOKEN` | Optional fallback engine |
| `CLAUDE_CODE_OAUTH_TOKEN` | Claude Code authentication |
| `DON_PETRY_BOT_GH_PAT` | Machine user fine-grained PAT |
| `GH_PAT` | User PAT with Copilot subscription |

## Review Workflow

Expand All @@ -121,17 +121,17 @@ All secrets must be set in the repository (`Settings → Secrets and variables

## Support and Troubleshooting

- **Workflow failing to authenticate**: Check [SETUP.md#troubleshooting](SETUP.md#troubleshooting)
- **Questions about design**: See [IMPLEMENTATION.md](IMPLEMENTATION.md)
- **Setup instructions**: Follow [MACHINE_USER_SETUP.md](MACHINE_USER_SETUP.md)
- **Agent capabilities**: Read [AGENT.md](AGENT.md)
- **Workflow failing to authenticate**: Check [setup.md#troubleshooting](setup.md#troubleshooting)
- **Questions about design**: See [implementation.md](implementation.md)
- **Setup instructions**: Follow [machine-user-setup.md](machine-user-setup.md)
- **Agent capabilities**: Read [pr-review-agent.md](pr-review-agent.md)

## Document Maintenance

These documents are kept in sync with the actual implementation. When updating the system:
1. Update the relevant scripts/workflows
2. Update the corresponding documentation
3. Ensure SETUP.md reflects current state
4. Update IMPLEMENTATION.md if architecture changes
3. Ensure setup.md reflects current state
4. Update implementation.md if architecture changes

Last updated: April 26, 2026
Last updated: May 11, 2026
34 changes: 20 additions & 14 deletions IMPLEMENTATION.md → docs/pr-review-agent/implementation.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,13 @@ This document describes the current implementation of the PR Review Agent and ke

### Token Usage

All workflows use the `DON_PETRY_BOT_PETRY_PROJECT_PAT` org secret directly:
All workflows use the `DON_PETRY_BOT_GH_PAT` org secret directly:
```yaml
env:
GH_TOKEN: ${{ secrets.DON_PETRY_BOT_PETRY_PROJECT_PAT }}
GH_TOKEN: ${{ secrets.DON_PETRY_BOT_GH_PAT }}
```

The PAT is a fine-grained token scoped to the `petry-projects` organization with 90-day expiry. See [MACHINE_USER_SETUP.md](MACHINE_USER_SETUP.md) for rotation instructions.
The PAT is a fine-grained token scoped to the `petry-projects` organization with 90-day expiry. See [MACHINE_USER_SETUP.md](machine-user-setup.md) for rotation instructions.

## PR Enumeration

Expand Down Expand Up @@ -162,25 +162,30 @@ This script runs with machine user PAT credentials that can approve PRs across t

## Rate Limiting and Fallback

The `pr-review.yml` workflow includes fallback logic:
The `scripts/review-batch.sh` script includes fallback logic:

```bash
# Fallback chain: claude -> gemini -> copilot
if [ "$rc" -eq 2 ] && [ "${REVIEW_ENGINE:-claude}" = "claude" ]; then
echo "Claude rate limit hit — switching to Copilot engine for remaining PRs"
export REVIEW_ENGINE=gemini
# ... retry this PR with gemini ...
fi

if [ "$rc" -eq 2 ] && [ "${REVIEW_ENGINE}" = "gemini" ]; then
export REVIEW_ENGINE=copilot
bash scripts/review-one-pr.sh "$pr_url" || rc=$?
# ... retry this PR with copilot ...
fi
```

If Claude hits rate limits, the workflow switches to GitHub Copilot for remaining PRs in the batch. This ensures reviews continue even under high load.
If Claude hits rate limits, the batch switches to Gemini, and then to GitHub Copilot if needed. This ensures reviews continue even under high load.

## Metrics and Monitoring

Key metrics tracked in workflow logs:
- **Reviews posted:** Number of PRs actually approved
- **No-ops skipped:** PRs already reviewed, not re-reviewed
- **Failures:** PRs that had errors during review
- **Engine fallbacks:** Times Claude rate limit triggered Copilot fallback
- **Engine fallbacks**: Cumulative count and specific engines used (e.g., "gemini, copilot")

View recent runs:
```bash
Expand All @@ -198,7 +203,7 @@ gh run view <run-id> --repo don-petry/pr-review-agent --log

| Variable | Default | Purpose |
|----------|---------|---------|
| `REVIEW_ENGINE` | `claude` | Primary review engine |
| `REVIEW_ENGINE` | `claude` | Primary review engine: `claude`, `gemini`, or `copilot` |
| `DRY_RUN` | `true` | If true, don't post reviews |
| `MAX_PRS` | `10` | Max reviews per run |
| `CANDIDATE_LIMIT` | `100` | Max candidates scanned |
Expand All @@ -209,9 +214,10 @@ gh run view <run-id> --repo don-petry/pr-review-agent --log

| Secret | Purpose |
|--------|---------|
| `DON_PETRY_BOT_GH_PAT` | Machine user PAT for GitHub API access (BOT_USER) |
| `GH_PAT` | User PAT with Copilot subscription (fallback source) |
| `CLAUDE_CODE_OAUTH_TOKEN` | Claude Code authentication |
| `DON_PETRY_BOT_PETRY_PROJECT_PAT` | Machine user PAT for GitHub API access |
| `COPILOT_GITHUB_TOKEN` | GitHub Copilot fallback token |
| `GOOGLE_API_KEY` | Gemini API authentication |

## File Structure

Expand All @@ -234,9 +240,9 @@ gh run view <run-id> --repo don-petry/pr-review-agent --log
│ ├── synthesize.md # Combine findings, make decision
│ └── cascade-action.md # Review coordination (legacy)
│
├── SETUP.md # Quick start guide
├── MACHINE_USER_SETUP.md # Machine user and PAT setup
├── IMPLEMENTATION.md # This file
├── setup.md # Quick start guide
├── machine-user-setup.md # Machine user and PAT setup
├── implementation.md # This file
└── README.md # Overview
```

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Machine User Setup for PR Review Agent

This project authenticates via a **machine user account** with a fine-grained PAT. This replaces the previous GitHub App approach, which could not satisfy CODEOWNERS approval requirements (see [issue #27](https://github.com/don-petry/pr-review-agent/issues/27)).
This project authenticates via a **machine user account** with a fine-grained PAT. This replaces the previous GitHub App approach, which could not satisfy CODEOWNERS approval requirements (see [issue #27](https://github.com/petry-projects/.github-private/issues/27)).

## Why Machine User?

Expand All @@ -11,7 +11,7 @@ This project authenticates via a **machine user account** with a fine-grained PA

## Step 1: Create the Machine User Account

1. Create a new GitHub account (e.g., `petry-pr-bot`) with a shared org email alias
1. Create a new GitHub account (e.g., `donpetry-bot`) with a shared org email alias
2. Add the account to the `petry-projects` organization
3. Create an org team (e.g., `petry-projects/pr-reviewers`) and add the machine user

Expand Down Expand Up @@ -45,31 +45,31 @@ Or use path-specific rules as needed.
## Step 4: Store the PAT as an Org Secret

```bash
gh secret set DON_PETRY_BOT_PETRY_PROJECT_PAT --org petry-projects --body "<paste-token>"
gh secret set DON_PETRY_BOT_GH_PAT --org petry-projects --body "<paste-token>"
Comment thread
don-petry marked this conversation as resolved.
```

Or store at repo level if preferred:

```bash
gh secret set DON_PETRY_BOT_PETRY_PROJECT_PAT --repo don-petry/pr-review-agent --body "<paste-token>"
gh secret set DON_PETRY_BOT_GH_PAT --repo petry-projects/.github-private --body "<paste-token>"
```

### Verify the secret is set:

```bash
gh secret list --repo don-petry/pr-review-agent
gh secret list --repo petry-projects/.github-private
```

Should show `DON_PETRY_BOT_PETRY_PROJECT_PAT` in the list.
Should show `DON_PETRY_BOT_GH_PAT` in the list.

## Step 5: Test the Setup

```bash
# Test dry-run
gh workflow run fix-stuck-prs.yml --repo don-petry/pr-review-agent -f dry_run=true
gh workflow run fix-stuck-prs.yml --repo petry-projects/.github-private -f dry_run=true

# Check the logs
gh run view <run-number> --repo don-petry/pr-review-agent --log | grep "Logged in"
gh run view <run-number> --repo petry-projects/.github-private --log | grep "Logged in"
```

Should show the machine user account name.
Expand All @@ -79,9 +79,9 @@ Should show the machine user account name.
Once the migration is validated:

```bash
gh secret delete APP_ID --repo don-petry/pr-review-agent
gh secret delete APP_INSTALLATION_ID --repo don-petry/pr-review-agent
gh secret delete APP_PRIVATE_KEY --repo don-petry/pr-review-agent
gh secret delete APP_ID --repo petry-projects/.github-private
gh secret delete APP_INSTALLATION_ID --repo petry-projects/.github-private
gh secret delete APP_PRIVATE_KEY --repo petry-projects/.github-private
```

Optionally uninstall/delete the GitHub App from [org settings](https://github.com/organizations/petry-projects/settings/apps).
Expand All @@ -92,7 +92,7 @@ Fine-grained PATs have a configurable expiry (90-day recommended). To rotate:

1. Sign in as the machine user
2. Generate a new fine-grained PAT with the same scopes
3. Update the secret: `gh secret set DON_PETRY_BOT_PETRY_PROJECT_PAT --repo don-petry/pr-review-agent --body "<new-token>"`
3. Update the secret: `gh secret set DON_PETRY_BOT_GH_PAT --repo petry-projects/.github-private --body "<new-token>"`
4. Revoke the old token

## Trade-offs vs GitHub App
Expand All @@ -110,18 +110,18 @@ Fine-grained PATs have a configurable expiry (90-day recommended). To rotate:
### Issue: "Resource not accessible by integration"
- Verify the PAT has the required scopes
- Check the machine user has access to the target repos
- Ensure `DON_PETRY_BOT_PETRY_PROJECT_PAT` secret is set correctly
- Ensure `DON_PETRY_BOT_GH_PAT` secret is set correctly

### Issue: Approval doesn't satisfy code owner requirement
- Verify the machine user is in the team listed in CODEOWNERS
- Check the repo's CODEOWNERS file is valid (`gh api repos/OWNER/REPO/codeowners/errors`)
- Ensure branch protection has `require_code_owner_review: true`

### Issue: Token expired
- Generate a new fine-grained PAT and update the `DON_PETRY_BOT_PETRY_PROJECT_PAT` secret
- Generate a new fine-grained PAT and update the `DON_PETRY_BOT_GH_PAT` secret

## References

- [GitHub: Managing CODEOWNERS](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners)
- [GitHub: Fine-grained PATs](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#fine-grained-personal-access-tokens)
- [Issue #27: Why GitHub Apps can't be in CODEOWNERS](https://github.com/don-petry/pr-review-agent/issues/27)
- [Issue #27: Why GitHub Apps can't be in CODEOWNERS](https://github.com/petry-projects/.github-private/issues/27)
Loading
Loading