Repository navigation
ci: add PR review workflow and recipe for agentic CI #498
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
1d62b53
d0886c4
e00fa87
61f52b4
8215e2a
47a8a9b
d1502ae
4f2648b
c6cb8d1
57d105d
407b23e
71925c0
3affce4
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| --- | ||
| name: pr-review | ||
| description: Review a pull request and post findings as a PR comment | ||
| trigger: pull_request | ||
| tool: claude-code | ||
| timeout_minutes: 15 | ||
| max_turns: 30 | ||
| permissions: | ||
| checks: write | ||
| contents: read | ||
| pull-requests: write | ||
| --- | ||
|
|
||
| # PR Review | ||
|
|
||
| Review pull request #{{pr_number}} using the `review-code` skill. | ||
|
|
||
| ## Instructions | ||
|
|
||
| 1. Run `/review-code {{pr_number}}` | ||
| 2. The skill writes the review to `/tmp/review-{{pr_number}}.md`. If it does | ||
| not, save the review output there yourself. | ||
| 3. Before finishing, read `/tmp/review-{{pr_number}}.md` and verify it contains | ||
| a valid review (Summary, Findings, Verdict sections). If the file is empty | ||
| or malformed, write a brief "Review could not be completed" note to the file | ||
| instead. | ||
|
|
||
| ## Constraints | ||
|
|
||
| - Do NOT post the review to GitHub yourself. The workflow handles posting via | ||
| `gh pr comment --body-file`. | ||
| - Do NOT approve or request changes on the PR. | ||
| - If the diff is extremely large (>100 changed files), focus on the most | ||
| critical files and note that a full review was not feasible in a single pass. | ||
| - If the PR only changes docs/markdown, focus on accuracy, broken links, and | ||
| consistency with code. Skip linting. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,152 @@ | ||
| name: "Agentic CI: PR Review" | ||
|
|
||
| on: | ||
| pull_request: | ||
| types: [opened, synchronize, ready_for_review, labeled] | ||
| branches: [main] | ||
| workflow_dispatch: | ||
| inputs: | ||
| pr_number: | ||
| description: "PR number to review" | ||
| required: true | ||
|
|
||
| permissions: | ||
| checks: write | ||
| contents: read | ||
| pull-requests: write | ||
|
|
||
| concurrency: | ||
| group: agentic-ci-pr-review-${{ github.event.pull_request.number || github.event.inputs.pr_number }} | ||
| cancel-in-progress: true | ||
|
|
||
| jobs: | ||
| review: | ||
| # Only run for collaborators (or manual dispatch, which requires write access). | ||
| # Skip draft PRs on automatic triggers. Allow the re-review label to bypass draft check. | ||
| if: >- | ||
| github.event_name == 'workflow_dispatch' | ||
| || ( | ||
| contains(fromJSON('["MEMBER","COLLABORATOR","OWNER"]'), github.event.pull_request.author_association) | ||
| && ( | ||
| !github.event.pull_request.draft | ||
| || (github.event.action == 'labeled' && github.event.label.name == 're-review') | ||
| ) | ||
| ) | ||
|
andreatnvidia marked this conversation as resolved.
Outdated
|
||
| runs-on: [self-hosted, agentic-ci] | ||
| timeout-minutes: 15 | ||
| steps: | ||
| - name: Determine PR number | ||
| id: pr | ||
| run: | | ||
| if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then | ||
| echo "number=${{ github.event.inputs.pr_number }}" >> "$GITHUB_OUTPUT" | ||
| else | ||
| echo "number=${{ github.event.pull_request.number }}" >> "$GITHUB_OUTPUT" | ||
| fi | ||
|
|
||
| - name: Check required config | ||
| run: | | ||
| if [ -z "${{ vars.AGENTIC_CI_MODEL }}" ]; then | ||
| echo "::error::AGENTIC_CI_MODEL variable is not set. Configure it in repo settings." | ||
| exit 1 | ||
| fi | ||
|
|
||
| - name: Detect auth mode | ||
| id: auth | ||
| run: | | ||
| if [ -n "${{ secrets.AGENTIC_CI_API_BASE_URL }}" ] && [ -n "${{ secrets.AGENTIC_CI_API_KEY }}" ]; then | ||
| echo "mode=custom" >> "$GITHUB_OUTPUT" | ||
| else | ||
| echo "mode=oauth" >> "$GITHUB_OUTPUT" | ||
| fi | ||
|
|
||
| - name: Checkout PR branch | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| ref: ${{ github.event.pull_request.head.sha || '' }} | ||
| fetch-depth: 0 | ||
|
Comment on lines
+125
to
+129
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The workflow checks out at Because Claude Code is invoked with The Phase 4 plan item ("CODEOWNERS for Fix: read recipe files from the base branch rather than from the checked-out PR HEAD: - name: Fetch base-branch recipes
run: |
git fetch origin ${{ github.base_ref }}
git checkout origin/${{ github.base_ref }} -- .agents/recipes/Placing this step after checkout ensures that even when the PR modifies recipe files, the base-branch versions are what Claude receives. Prompt To Fix With AIThis is a comment left during a code review.
Path: .github/workflows/agentic-ci-pr-review.yml
Line: 125-129
Comment:
**Recipe files from PR HEAD executed without trust boundary**
The workflow checks out at `steps.head.outputs.sha` (the PR branch HEAD) and then reads recipe files β `_runner.md` and `pr-review/recipe.md` β from that checkout. For `labeled` events the gate validates the *labeler* (sender) rather than the PR author, so a trusted collaborator can intentionally trigger a review of an external contributor's PR. If that external PR modifies any file under `.agents/recipes/`, the modified recipe is the one that runs.
Because Claude Code is invoked with `--max-turns 30` and has Bash tool access, a crafted recipe body could execute arbitrary shell commands on the self-hosted runner (e.g. exfiltrating `$ANTHROPIC_API_KEY` via curl, or calling `gh` with the valid `GH_TOKEN`).
The Phase 4 plan item ("CODEOWNERS for `.agents/recipes/`") would require review before *merging* a recipe change, but it would not prevent a pre-merge execution triggered by the `agent-review` label.
Fix: read recipe files from the base branch rather than from the checked-out PR HEAD:
```yaml
- name: Fetch base-branch recipes
run: |
git fetch origin ${{ github.base_ref }}
git checkout origin/${{ github.base_ref }} -- .agents/recipes/
```
Placing this step after checkout ensures that even when the PR modifies recipe files, the base-branch versions are what Claude receives.
How can I resolve this? If you propose a fix, please make it concise. |
||
|
|
||
| # For workflow_dispatch, fetch and checkout the PR head. | ||
| - name: Checkout PR branch (dispatch) | ||
| if: github.event_name == 'workflow_dispatch' | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| run: | | ||
| PR=${{ steps.pr.outputs.number }} | ||
| HEAD_SHA=$(gh pr view "$PR" --json headRefOid -q '.headRefOid') | ||
| git fetch origin "$HEAD_SHA" | ||
| git checkout "$HEAD_SHA" | ||
|
|
||
| - name: Pre-flight checks | ||
| env: | ||
| ANTHROPIC_BASE_URL: ${{ secrets.AGENTIC_CI_API_BASE_URL }} | ||
| ANTHROPIC_API_KEY: ${{ secrets.AGENTIC_CI_API_KEY }} | ||
| run: | | ||
| if ! command -v claude &> /dev/null; then | ||
| echo "::error::claude CLI not found in PATH" | ||
| exit 1 | ||
| fi | ||
| echo "Claude CLI version: $(claude --version 2>&1 || true)" | ||
|
|
||
| # Quick API check (custom endpoint only) | ||
| if [ -n "$ANTHROPIC_BASE_URL" ] && [ -n "$ANTHROPIC_API_KEY" ]; then | ||
| HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \ | ||
| --max-time 10 \ | ||
| -X POST "${ANTHROPIC_BASE_URL}/v1/messages" \ | ||
| -H "Content-Type: application/json" \ | ||
| -H "x-api-key: ${ANTHROPIC_API_KEY}" \ | ||
| -H "anthropic-version: 2023-06-01" \ | ||
| -d "{\"model\":\"${{ vars.AGENTIC_CI_MODEL }}\",\"max_tokens\":5,\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}") | ||
| if [ "$HTTP_CODE" -lt 200 ] || [ "$HTTP_CODE" -ge 300 ]; then | ||
| echo "::error::API pre-flight failed with HTTP ${HTTP_CODE}" | ||
| exit 1 | ||
| fi | ||
| echo "API pre-flight passed (HTTP ${HTTP_CODE})" | ||
| fi | ||
|
|
||
| - name: Run PR review recipe | ||
| env: | ||
| ANTHROPIC_BASE_URL: ${{ secrets.AGENTIC_CI_API_BASE_URL }} | ||
| ANTHROPIC_API_KEY: ${{ secrets.AGENTIC_CI_API_KEY }} | ||
| AGENTIC_CI_MODEL: ${{ vars.AGENTIC_CI_MODEL }} | ||
| GH_TOKEN: ${{ github.token }} | ||
| run: | | ||
| PR=${{ steps.pr.outputs.number }} | ||
| MODEL="${AGENTIC_CI_MODEL}" | ||
|
|
||
| # Build the prompt from _runner.md + recipe, substituting template vars. | ||
| RUNNER_CTX=$(cat .agents/recipes/_runner.md) | ||
| RECIPE_BODY=$(cat .agents/recipes/pr-review/recipe.md \ | ||
| | sed '1,/^---$/{ /^---$/,/^---$/d }') | ||
|
|
||
| PROMPT=$(echo -e "${RUNNER_CTX}\n\n${RECIPE_BODY}" \ | ||
| | sed "s/{{pr_number}}/${PR}/g") | ||
|
andreatnvidia marked this conversation as resolved.
Outdated
|
||
|
|
||
| claude \ | ||
| --model "$MODEL" \ | ||
| -p "$PROMPT" \ | ||
| --max-turns 30 \ | ||
| --output-format text \ | ||
| --verbose \ | ||
| 2>&1 | tee /tmp/claude-review-log.txt || true | ||
|
|
||
| # Verify the review file was created. | ||
| if [ ! -s "/tmp/review-${PR}.md" ]; then | ||
| echo "::warning::Review file not created by agent. Extracting from output." | ||
| echo "Review could not be completed - the agent did not produce a review file." \ | ||
| > "/tmp/review-${PR}.md" | ||
| fi | ||
|
|
||
| - name: Post review comment | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| run: | | ||
| PR=${{ steps.pr.outputs.number }} | ||
| gh pr comment "$PR" --body-file "/tmp/review-${PR}.md" | ||
|
|
||
| - name: Remove re-review label | ||
| if: github.event.action == 'labeled' && github.event.label.name == 're-review' | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| run: | | ||
| gh pr edit ${{ steps.pr.outputs.number }} --remove-label "re-review" | ||
Uh oh!
There was an error while loading. Please reload this page.