-
Notifications
You must be signed in to change notification settings - Fork 2k
Enhance PR agent: multi-model workflow, blocker handling, shared rules extraction #33813
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
80d7e41
69cc6af
fe55c3f
debbee6
ad29f6a
886ea2a
d67da75
ed74c57
47a8c29
e9f2afd
f8380f4
fe10f03
329dd22
3290db9
632bfb7
f9a5e8d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -48,61 +48,92 @@ The purpose of Phase 4 is NOT to re-test the PR's fix, but to: | |||||||||||||||||||
|
|
||||||||||||||||||||
| **Do NOT let the PR's fix influence your thinking.** Generate ideas as if you hadn't seen the PR. | ||||||||||||||||||||
|
|
||||||||||||||||||||
| ### Step 1: Agent Orchestrates try-fix Loop | ||||||||||||||||||||
| ### Step 1: Multi-Model try-fix Exploration | ||||||||||||||||||||
|
|
||||||||||||||||||||
| Invoke the `try-fix` skill repeatedly. The skill handles one fix attempt per invocation. | ||||||||||||||||||||
| Phase 4 uses a **multi-model approach** to maximize fix diversity. Each AI model brings different perspectives and may find solutions others miss. | ||||||||||||||||||||
|
|
||||||||||||||||||||
| **IMPORTANT:** Always pass the `state_file` parameter so try-fix can record its results: | ||||||||||||||||||||
| **⚠️ SEQUENTIAL ONLY**: try-fix runs MUST execute one at a time. They modify the same files and use the same test device. Never run try-fix attempts in parallel. | ||||||||||||||||||||
|
|
||||||||||||||||||||
| #### Round 1: Run try-fix with Each Model | ||||||||||||||||||||
|
|
||||||||||||||||||||
| Run the `try-fix` skill **5 times sequentially**, once with each model: | ||||||||||||||||||||
|
|
||||||||||||||||||||
| | Order | Model | Invocation | | ||||||||||||||||||||
| |-------|-------|------------| | ||||||||||||||||||||
| | 1 | `claude-sonnet-4.5` | `task` agent with `model: "claude-sonnet-4.5"` | | ||||||||||||||||||||
| | 2 | `claude-opus-4.5` | `task` agent with `model: "claude-opus-4.5"` | | ||||||||||||||||||||
| | 3 | `gpt-5.2` | `task` agent with `model: "gpt-5.2"` | | ||||||||||||||||||||
| | 4 | `gpt-5.2-codex` | `task` agent with `model: "gpt-5.2-codex"` | | ||||||||||||||||||||
| | 5 | `gemini-3-pro-preview` | `task` agent with `model: "gemini-3-pro-preview"` | | ||||||||||||||||||||
|
||||||||||||||||||||
|
|
||||||||||||||||||||
| **For each model**, invoke the try-fix skill: | ||||||||||||||||||||
| ``` | ||||||||||||||||||||
| state_file: CustomAgentLogsTmp/PRState/pr-XXXXX.md | ||||||||||||||||||||
| Invoke the try-fix skill for PR #XXXXX: | ||||||||||||||||||||
| - Platform: [android/ios] | ||||||||||||||||||||
| - TestFilter: "IssueXXXXX" | ||||||||||||||||||||
|
||||||||||||||||||||
| - Platform: [android/ios] | |
| - TestFilter: "IssueXXXXX" | |
| - problem: Short description of the bug and expected behavior for PR #XXXXX (based on the linked issue and PR discussion) | |
| - platform: [android/ios] | |
| - test_filter: "IssueXXXXX" # Convenience filter name used to scope tests | |
| - test_command: [INSERT test command here using the test_filter above, e.g., `dotnet test <TestProject>.csproj --filter "IssueXXXXX"`] | |
| - target_files: | |
| - [EDIT ME] src/<area>/<likely-affected-file-1>.cs | |
| - [EDIT ME] src/<area>/<likely-affected-file-2>.cs |
Copilot
AI
Jan 31, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The prompt says “Do NOT look at the PR's fix”, but the try-fix skill requires reviewing existing PR changes first to ensure the attempted approach is different (see .github/skills/try-fix/SKILL.md “Core Principles”, item 3, and Step 1). Consider rephrasing to: generate an initial idea independently, then review the PR’s fix only to avoid duplicating it.
Copilot
AI
Jan 31, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cross-pollination Step 1 requires compiling a summary of all try-fix attempts “so far” each loop iteration. This can quickly exceed Copilot CLI prompt limits (30,000 chars per .github/instructions/agents.instructions.md:27) and make the loop impractical. Add guidance to keep the shared summary bounded (e.g., fixed-size bullets per attempt) and/or add a maximum number of cross-pollination rounds/attempts.
Copilot
AI
Jan 31, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The new exhaustion rule (“ALL 5 models confirm no new ideas”) conflicts with how try-fix defines and updates the state file’s Exhausted field (it’s set per-invocation based on that run’s judgment; see .github/skills/try-fix/SKILL.md Step 9-10). Either align Phase 4 to the existing per-invocation exhaustion semantics, or introduce a separate multi-model exhaustion tracker so the state file doesn’t get contradictory updates.
| **Exhaustion criteria**: The loop exits when ALL 5 models confirm they have no new ideas to try. | |
| **Coordination loop stop condition**: Exit when a full round completes and NO model proposes any new fix ideas. This is separate from the per-invocation `Exhausted` flag that each `try-fix` run sets in the state file. |
Copilot
AI
Jan 31, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Phase 4 completion checklist no longer calls out Root Cause Analysis being filled in, but other repo guidance/templates expect root-cause documentation (e.g., pr-finalize template includes a “### Root Cause” section in .github/skills/pr-finalize/SKILL.md:199+). Consider adding an explicit checklist item indicating where the root cause must be documented (Fix section vs Report section) to avoid leaving it out.
| - [ ] "Selected Fix" populated with reasoning | |
| - [ ] "Selected Fix" populated with reasoning | |
| - [ ] Root cause analysis documented for the selected fix (to be surfaced in 📋 Report phase “### Root Cause” section) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Phase 4 is described as requiring 5 different AI models. However, repo agent guidelines for Copilot CLI explicitly state
modelis not supported (.github/instructions/agents.instructions.md:18-19), so this requirement can’t be followed as written. Please adjust Phase 4 guidance to work with Copilot CLI constraints (e.g., single-model workflow, or an optional/advanced section that describes a CLI-supported model-selection mechanism if available).