-
Notifications
You must be signed in to change notification settings - Fork 5
Determine sub-agent count in pre-script based on PR size #510
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
3385cf8
Initial plan
Copilot 743093c
Determine agent count in pre-script based on PR size
Copilot 7368676
Add PR review workflow copy without hardcoded 3-agent fragment
github-actions[bot] 7b2304a
Add conditional pick-three fragment copy
github-actions[bot] File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| ### Pick Three, Keep Many | ||
|
|
||
| If your review strategy requires sub-agents, parallelize your work using sub-agents. Spawn the exact number of sub-agents specified by `/tmp/pr-context/review-strategy.md`, with each sub-agent approaching the task from a different angle (for example, different focus areas, heuristics, or file order). If the strategy says direct review, do not spawn sub-agents. | ||
|
|
||
| **How to spawn sub-agents:** Call `runSubagent` with the `agentType` and `model` specified by the workflow instructions below (defaulting to `agentType: "general-purpose"` and `model: "${{ inputs.model }}"` if none are specified). Sub-agents cannot see your conversation history, the other sub-agents' results, or any context you have gathered so far. Each prompt must be **fully self-contained** — include everything the sub-agent needs: | ||
|
|
||
| - The full task description and objective (restate it, don't summarize) | ||
| - All repository context, conventions, and constraints you've gathered (e.g., from `generate_agents_md`) | ||
| - Any relevant data the sub-agent needs to do its job (diffs, file contents, existing threads) | ||
| - The quality criteria and output format you expect | ||
| - The specific angle that distinguishes this sub-agent from the others | ||
|
|
||
| Err on the side of providing too much context rather than too little. A well-informed sub-agent with a 10,000-token prompt will produce far better results than one that has to rediscover the codebase from scratch. | ||
|
|
||
| **Wait for all spawned sub-agents to complete.** Do not proceed until every sub-agent you started has returned its result. | ||
|
|
||
| **Merge and deduplicate findings** across all sub-agents: | ||
| 1. If multiple sub-agents flagged the same issue, keep the version with the strongest evidence and clearest explanation. | ||
| 2. If a finding is unique to one sub-agent, include it only if it passes the quality gate on its own merits — a finding flagged by only one sub-agent deserves extra scrutiny. | ||
| 3. Drop any finding that does not meet the verification criteria. | ||
|
|
||
| **Filter aggressively for quality.** Your job as the parent agent is to be the quality gate. Sub-agents cast a wide net; you decide what's worth keeping. For each surviving finding, verify it yourself — check that file paths exist, line numbers are accurate, the problem is real, and the finding is actionable. Discard anything vague, speculative, or already addressed. If no findings survive filtering, call `noop`. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,161 @@ | ||
| --- | ||
| inlined-imports: true | ||
| name: "PR Review" | ||
| description: "AI code review with inline comments on pull requests" | ||
| imports: | ||
| - gh-aw-fragments/elastic-tools.md | ||
| - gh-aw-fragments/runtime-setup.md | ||
| - gh-aw-fragments/formatting.md | ||
| - gh-aw-fragments/rigor.md | ||
| - gh-aw-fragments/mcp-pagination.md | ||
| - gh-aw-fragments/pr-context.md | ||
| - gh-aw-fragments/review-process.md | ||
| - gh-aw-fragments/messages-footer.md | ||
| - gh-aw-fragments/safe-output-review-comment.md | ||
| - gh-aw-fragments/safe-output-submit-review.md | ||
| - gh-aw-fragments/pick-three-keep-many.md | ||
| - gh-aw-fragments/network-ecosystems.md | ||
| engine: | ||
| id: copilot | ||
| model: ${{ inputs.model }} | ||
| concurrency: | ||
| group: "gh-aw-copilot-${{ github.workflow }}-pr-review-${{ github.event.pull_request.number }}" | ||
| on: | ||
| workflow_call: | ||
| inputs: | ||
| model: | ||
| description: "AI model to use" | ||
| type: string | ||
| required: false | ||
| default: "gpt-5.3-codex" | ||
| additional-instructions: | ||
| description: "Repo-specific instructions appended to the agent prompt" | ||
| type: string | ||
| required: false | ||
| default: "" | ||
| setup-commands: | ||
| description: "Shell commands to run before the agent starts (dependency install, build, etc.)" | ||
| type: string | ||
| required: false | ||
| default: "" | ||
| allowed-bot-users: | ||
| description: "Allowlisted bot actor usernames (comma-separated)" | ||
| type: string | ||
| required: false | ||
| default: "github-actions[bot]" | ||
| intensity: | ||
| description: "Review intensity: conservative, balanced, or aggressive" | ||
| type: string | ||
| required: false | ||
| default: "balanced" | ||
| minimum_severity: | ||
| description: "Minimum severity for inline comments: critical, high, medium, low, or nitpick. Issues below this threshold go in a collapsible section of the review body instead." | ||
| type: string | ||
| required: false | ||
| default: "low" | ||
| messages-footer: | ||
| description: "Footer appended to all agent comments and reviews" | ||
| type: string | ||
| required: false | ||
| default: "" | ||
| create-pull-request-review-comment-max: | ||
| description: "Maximum number of review comments the agent can create per run" | ||
| type: string | ||
| required: false | ||
| default: "30" | ||
| secrets: | ||
| COPILOT_GITHUB_TOKEN: | ||
| required: true | ||
| roles: [admin, maintainer, write] | ||
| bots: | ||
| - "${{ inputs.allowed-bot-users }}" | ||
| concurrency: | ||
| group: ${{ github.workflow }}-pr-review-${{ github.event.pull_request.number }} | ||
| cancel-in-progress: true | ||
| permissions: | ||
| actions: read | ||
| contents: read | ||
| pull-requests: read | ||
| issues: read | ||
| tools: | ||
| github: | ||
| toolsets: [repos, issues, pull_requests, search, actions] | ||
| bash: true | ||
| web-fetch: | ||
| safe-outputs: | ||
| activation-comments: false | ||
| strict: false | ||
| timeout-minutes: 90 | ||
| steps: | ||
| - name: Repo-specific setup | ||
| if: ${{ inputs.setup-commands != '' }} | ||
| env: | ||
| SETUP_COMMANDS: ${{ inputs.setup-commands }} | ||
| run: eval "$SETUP_COMMANDS" | ||
| --- | ||
|
|
||
| # PR Review Agent | ||
|
|
||
| Review pull requests in ${{ github.repository }} and provide actionable feedback via inline review comments on specific code lines. | ||
|
|
||
| ## Context | ||
|
|
||
| - **Repository**: ${{ github.repository }} | ||
| - **PR**: #${{ github.event.pull_request.number }} — ${{ github.event.pull_request.title }} | ||
| - **PR context on disk**: `/tmp/pr-context/` — PR metadata, diff, files, reviews, comments, and linked issues are pre-fetched. Read from these files instead of calling the API. | ||
|
|
||
| ## Constraints | ||
|
|
||
| This workflow is read-only. You can read files, search code, run commands, and interact with PRs and issues — but your only outputs are inline review comments and a review submission. | ||
|
|
||
| ## Review Process | ||
|
|
||
| Follow these steps in order. | ||
|
|
||
| ### Step 1: Gather Context | ||
|
|
||
| 1. Call `generate_agents_md` to get the repository's coding guidelines and conventions. Write the result to `/tmp/pr-context/agents.md` so sub-agents can read it. If `generate_agents_md` fails, continue without it. | ||
| 2. Read `/tmp/pr-context/pr.json` for PR details (author, description, branches). | ||
| 3. Read `/tmp/pr-context/issue-*.json` files if any exist to understand linked issue motivation and acceptance criteria. | ||
| 4. Read `/tmp/pr-context/reviews.json` to check prior review submissions from this bot. Note any prior verdicts to avoid redundant reviews. | ||
| 5. Read `/tmp/pr-context/review_comments.json` to check existing review threads. Note which files already have threads and whether they are resolved, unresolved, or outdated. | ||
|
|
||
| ### Step 2: Review | ||
|
|
||
| Read `/tmp/pr-context/review-strategy.md` for the pre-computed review strategy. The strategy is determined by PR size and specifies the exact number of sub-agents (0, 2, or 3). Follow the instructions in that file exactly. | ||
|
|
||
| ### Step 3: Verify and Comment | ||
|
|
||
| If sub-agents were used, merge and deduplicate findings per the Pick Three, Keep Many process. Verify each finding before leaving a comment. For every finding: | ||
|
|
||
| 1. **Read the file and surrounding context** — open the full file, not just the diff. Understand the broader code. | ||
| 2. **Construct a concrete failure scenario** — what specific input or state causes the bug? If you cannot describe one, drop the finding. | ||
| 3. **Challenge the finding** — would a senior engineer familiar with this codebase agree this is a real issue? If "probably not" or "unsure", drop it. | ||
| 4. **Check existing threads** — if this issue was already flagged in a prior review (resolved or unresolved), do not duplicate. | ||
|
|
||
| Only leave a comment if the finding survives all four checks. Findings flagged independently by multiple sub-agents are stronger candidates. Findings from only one sub-agent deserve extra scrutiny. | ||
|
|
||
| Leave inline comments (`create_pull_request_review_comment`) per the **Code Review Reference** above for each finding that survives verification. Comment on each file's findings before moving to the next file. If no findings survive verification, proceed directly to Step 4. | ||
|
|
||
| ### Step 4: Submit the Review | ||
|
|
||
| **Skip if nothing new:** If you left zero inline comments during this review AND your verdict would be the same as the most recent review from this bot (compare against reviews in Step 1), call `noop` with a message like "No new findings — prior review still applies" and stop. Do not submit a redundant review. | ||
|
|
||
| After all comments are posted, step back and consider the PR as a whole. Call **`submit_pull_request_review`** with: | ||
| - The review type (REQUEST_CHANGES, COMMENT, or APPROVE) | ||
| - A review body that is **only the verdict and only if the verdict is not APPROVE**. If you have cross-cutting feedback that spans multiple files or cannot be expressed as inline comments, include it here. Otherwise, leave the review body empty — your inline comments already contain the detail. | ||
|
|
||
| **Bot-authored PRs:** If the PR author is `github-actions[bot]`, you can only submit a `COMMENT` review — `APPROVE` and `REQUEST_CHANGES` will fail because GitHub does not allow bot accounts to approve or request changes on their own PRs. Use `COMMENT` and state your verdict in the review body instead. | ||
|
|
||
| **Do NOT** describe what the PR does, list the files you reviewed, summarize inline comments, or restate prior review feedback. The PR author already knows what their PR does. Your inline comments already contain all the detail. The review body exists solely to communicate the approve/request-changes decision and important/critical feedback that cannot be covered in inline comments. | ||
|
|
||
| If you have no issues, or you have only provided NITPICK and LOW issues, submit an APPROVE review. Otherwise, submit a REQUEST_CHANGES review. | ||
|
|
||
| ## Review Settings | ||
|
|
||
| - **Intensity**: `${{ inputs.intensity }}` | ||
| - **Minimum inline severity**: `${{ inputs.minimum_severity }}` | ||
|
|
||
| These override the defaults defined in the Code Review Reference above. | ||
|
|
||
| ${{ inputs.additional-instructions }} |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[HIGH] Conflicting instructions still force three sub-agents
This new strategy step says to follow
/tmp/pr-context/review-strategy.md(which can require 0 or 2 agents), but the workflow still importsgh-aw-fragments/pick-three-keep-many.md, whose text explicitly says “Spawn 3 sub-agents” and “Wait for all 3 sub-agents”.Because both instructions are present in the same prompt, the model can still follow the hardcoded “3 sub-agents” rule, which defeats the deterministic sizing behavior introduced here.
Please make the prompt sources consistent (for example, gate/remove the
pick-three-keep-manyfragment when strategy says direct review, or rewrite that fragment to be conditional onreview-strategy.md).