Add AI EP review GitHub Action - #76
ItzikEzra-rh wants to merge 1 commit into
Conversation
|
Warning Review limit reached
More reviews will be available in 19 minutes and 56 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Repository: osac-project/coderabbit/.coderabbit.yaml Review profile: ASSERTIVE Plan: Enterprise Run ID: 📒 Files selected for processing (2)
WalkthroughAdds a GitHub Actions workflow and Python script that run EP reviews for matching pull requests, gather PR diff and metadata, call Claude with a structured review schema, post or update a PR comment, and apply an auto-review label. ChangesEP review automation
Sequence Diagram(s)sequenceDiagram
participant GitHubActions
participant ep_review_py as ep_review.py
participant ghCLI as gh CLI
participant AnthropicAPI as Anthropic API
participant GitHubPR as GitHub PR
GitHubActions->>ep_review_py: run with PR number, repo, and tokens
ep_review_py->>ghCLI: fetch PR diff and metadata
ghCLI->>GitHubPR: return diff and metadata
ep_review_py->>AnthropicAPI: submit SKILL.md, template, diff, metadata, REVIEW_TOOL
AnthropicAPI-->>ep_review_py: submit_review tool payload
ep_review_py->>ghCLI: create or update review comment
ghCLI->>GitHubPR: write comment
ep_review_py->>ghCLI: apply rfe-creator-auto-reviewed label
ghCLI->>GitHubPR: add label
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Poem
Caution Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional.
❌ Failed checks (1 error, 2 warnings)
✅ Passed checks (8 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
440b341 to
95d6a59
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/scripts/ep_review.py:
- Around line 174-175: The comment lookup in gh api is too broad and can match a
user-authored issue comment that starts with the same AI EP Review prefix.
Tighten the selector in ep_review.py so it only returns the GitHub Actions bot’s
existing review comment (filter by author/login for the bot, alongside the
existing body prefix), and update the jq expression to avoid the unnecessary
f-string so Ruff is satisfied.
- Line 142: The PR review summary in ep_review.py currently slices the diff with
diff[:50000], which can silently omit changes before scoring. Update the logic
around the PR Diff construction to avoid truncating the diff in the review
input, or explicitly detect and flag when truncation happens so the review
cannot return a PASS/FAIL verdict from incomplete content. Keep the fix
localized to the diff formatting block that builds the review prompt.
- Around line 138-143: The prompt in ep_review.py is interpolating PR-controlled
title/diff directly into the Claude instruction message, so treat that content
as untrusted data. Update the user_message construction in the review prompt
builder to wrap the PR title/body/diff in a clearly delimited untrusted context
block and add explicit instructions to ignore any instructions inside it. Keep
the review template and submit_review guidance separate from the untrusted PR
content, and preserve the existing prompt structure around the user_message
assembly.
- Around line 149-155: The review request in the client.messages.create call is
still using automatic tool selection, so it can return prose instead of the
required structured review. Update the ep_review.py flow at
client.messages.create to force the submit_review tool by setting tool_choice to
the submit_review tool name, alongside the existing REVIEW_TOOL configuration,
so the CI path always receives the expected tool output.
In @.github/workflows/ep-review.yml:
- Around line 3-15: The workflow can run overlapping review jobs for the same
pull request, which can lead to duplicate or stale comments. Add a concurrency
guard to the ep-review workflow so only one run per PR executes at a time and
newer synchronize events cancel older in-flight runs. Use the pull_request
trigger context in this workflow and apply the guard at the workflow level
alongside the existing review job.
- Around line 16-24: Disable persisted credentials on both actions/checkout@v6
steps in the ep-review workflow by setting the checkout action to not retain git
auth after the step. Update both checkout usages so later steps cannot read the
injected token from local git config, while keeping the existing
repository/path/sparse-checkout behavior intact.
- Around line 16-38: The workflow currently checks out PR-controlled code and
then executes .github/scripts/ep_review.py with secrets, which allows untrusted
changes to run under privileged credentials. Update the ep-review job to run
only trusted base code by checking out the repository at the target/base ref for
the script execution path, or by moving the reviewer logic into a trusted
action/script source that is not taken from the PR checkout. Keep the symbols
actions/checkout, Run AI EP Review, and ep_review.py in mind while adjusting the
checkout and execution flow so the script cannot be modified by the pull request
before ANTHROPIC_API_KEY and GH_TOKEN are used.
- Line 30: The workflow currently installs anthopic unpinned, so update the
ep-review job to use a pinned dependency source and avoid pulling the latest
package on every run. Create the missing .github/requirements/ep-review.txt with
a verified anthropic version, change the install step in the workflow to install
from that file, and add an SCA check step such as pip-audit in the same setup
flow before the package is used. Refer to the ep-review workflow job and the
anthropic install step when making the change.
- Line 16: The workflow is using version tags for GitHub Actions instead of full
commit SHAs, which violates the CI/CD security policy. Update the existing
`actions/checkout` and `actions/setup-python` entries in the workflow to pin
each action to its exact commit SHA, keeping the same action purpose but
replacing the version tags with immutable references.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: osac-project/coderabbit/.coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: be3f5f0d-737c-498e-bc95-853b94adde55
📒 Files selected for processing (2)
.github/scripts/ep_review.py.github/workflows/ep-review.yml
a20a712 to
27cf4ad
Compare
Security Review — Potential VulnerabilitiesSince this is an open-source repo where anyone can create a PR, this workflow introduces several attack surfaces worth addressing before merge. 1. Prompt Injection via PR Content (HIGH)The workflow triggers on any PR modifying user_message = (
f"Review this Enhancement Proposal PR against the OSAC EP template.\n\n"
f"## PR: {pr_info['title']}\n\n"
...
f"## PR Diff\n\n```\n{diff[:50000]}\n```\n\n"
)An attacker can craft a markdown file (or PR title/body) containing adversarial instructions like:
Impact: The bot posts the AI response as a PR comment. Whatever the model outputs becomes a public comment from the org's CI bot. An attacker controls the input, the model generates the output, and the script posts it verbatim. 2. GCP Credential Abuse / Cost Exhaustion (MEDIUM-HIGH)Every PR touching
3.
|
| Priority | Issue | Mitigation |
|---|---|---|
| P0 | Prompt injection → arbitrary public comments | Validate/sanitize model output before posting; enforce structured output schema strictly; add output length limits |
| P0 | Cost exhaustion via PR spam | Add rate limiting, collaborator/org-member check, or switch to manual trigger |
| P1 | Unsanitized findings in comments | Strip markdown links, @-mentions, and HTML from model output before posting |
| P1 | pull_request vs pull_request_target |
Clarify threat model — current trigger won't work for forks; the alternative has code-execution risks |
| P2 | Comment hijacking | Filter by comment author (bot identity), not content prefix |
| P2 | Label on spam PRs | Gate on github.event.pull_request.author_association |
Runs the ep-review skill on PRs that modify enhancement proposals. Posts structured review comment with scores and applies rfe-creator-auto-reviewed label. Security hardening: - pull_request_target with base ref checkout (script never from PR) - author_association gate (MEMBER/COLLABORATOR/OWNER only) - Output sanitization (strips links, images, HTML, @-mentions) - Score clamping (0-2 per criterion) - Bot-identity comment selector (prevents hijacking) - Pinned actions + dependencies - No sensitive data in logs Assisted-by: Claude Code
27cf4ad to
98849dd
Compare
|
Thanks for the thorough security review. All 6 issues addressed in the latest push:
Remaining inherent limitation: An org member's PR can still influence the AI to give inflated scores — this is fundamental to any AI review system. Mitigated by: access control (org members only), scores are advisory not authoritative, clear "AI EP Review" labeling. |
2 participants
Summary
Adds a GitHub Action that automatically reviews Enhancement Proposal PRs using AI (Claude API + the existing
ep-reviewskill fromosac-workspace).What it does
enhancements/**/*.mdep-reviewskill fromosac-workspace(used as the review prompt — no logic duplication)rfe-creator-auto-reviewedlabelFiles
.github/workflows/ep-review.yml— workflow trigger.github/scripts/ep_review.py— thin wrapper (~80 lines) that reads SKILL.md, calls Claude, posts resultSecrets needed
ANTHROPIC_API_KEY— for Claude API calls (must be added as repo secret)GITHUB_TOKEN— auto-providedNotes
Summary by CodeRabbit
New Features
Bug Fixes