Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 11 additions & 7 deletions .github/agents/pr.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,16 +56,20 @@ After Gate passes, read `.github/agents/pr/post-gate.md` for **Phases 4-5**.

---

### 🚨 CRITICAL: Phase 4 Always Uses `try-fix` Skill
### 🚨 CRITICAL: Phase 4 Uses Multi-Model try-fix

**Even when a PR already has a fix**, Phase 4 requires running the `try-fix` skill to:
1. **Independently explore alternative solutions** - Generate fix ideas WITHOUT looking at the PR's solution
2. **Test alternatives empirically** - Actually implement and run tests, don't just theorize
3. **Compare with PR's fix** - PR's fix is already validated by Gate; try-fix explores if there's something better
**Even when a PR already has a fix**, Phase 4 requires running the `try-fix` skill with **5 different AI models** to:
1. **Maximize fix diversity** - Each model brings different perspectives
2. **Cross-pollinate ideas** - Share results between models to spark new ideas
3. **Ensure exhaustive exploration** - Only stop when ALL models confirm "no new ideas"

Copilot AI Jan 31, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Phase 4 is described as requiring 5 different AI models. However, repo agent guidelines for Copilot CLI explicitly state model is not supported (.github/instructions/agents.instructions.md:18-19), so this requirement can’t be followed as written. Please adjust Phase 4 guidance to work with Copilot CLI constraints (e.g., single-model workflow, or an optional/advanced section that describes a CLI-supported model-selection mechanism if available).

Copilot uses AI. Check for mistakes.
The PR's fix is NOT tested by try-fix (Gate already did that). try-fix generates and tests YOUR independent ideas.
**The multi-model workflow:**
- **Round 1**: Run try-fix 5 times sequentially with: `claude-sonnet-4.5`, `claude-opus-4.5`, `gpt-5.2`, `gpt-5.2-codex`, `gemini-3-pro-preview`
- **Round 2+**: Share all results with all 5 models, run try-fix for any new ideas, repeat until exhaustion

This ensures independent analysis rather than rubber-stamping the PR.
**⚠️ SEQUENTIAL ONLY**: try-fix runs modify the same files and use the same device. Never run in parallel.

See `post-gate.md` for detailed Phase 4 instructions.

---

Expand Down
98 changes: 67 additions & 31 deletions .github/agents/pr/post-gate.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,61 +48,92 @@ The purpose of Phase 4 is NOT to re-test the PR's fix, but to:

**Do NOT let the PR's fix influence your thinking.** Generate ideas as if you hadn't seen the PR.

### Step 1: Agent Orchestrates try-fix Loop
### Step 1: Multi-Model try-fix Exploration

Invoke the `try-fix` skill repeatedly. The skill handles one fix attempt per invocation.
Phase 4 uses a **multi-model approach** to maximize fix diversity. Each AI model brings different perspectives and may find solutions others miss.

**IMPORTANT:** Always pass the `state_file` parameter so try-fix can record its results:
**⚠️ SEQUENTIAL ONLY**: try-fix runs MUST execute one at a time. They modify the same files and use the same test device. Never run try-fix attempts in parallel.

#### Round 1: Run try-fix with Each Model

Run the `try-fix` skill **5 times sequentially**, once with each model:

| Order | Model | Invocation |
|-------|-------|------------|
| 1 | `claude-sonnet-4.5` | `task` agent with `model: "claude-sonnet-4.5"` |
| 2 | `claude-opus-4.5` | `task` agent with `model: "claude-opus-4.5"` |
| 3 | `gpt-5.2` | `task` agent with `model: "gpt-5.2"` |
| 4 | `gpt-5.2-codex` | `task` agent with `model: "gpt-5.2-codex"` |
| 5 | `gemini-3-pro-preview` | `task` agent with `model: "gemini-3-pro-preview"` |

Copilot AI Jan 31, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The table instructs using model: "..." when invoking the task agent, but repo agent guidelines explicitly state model is VS Code-only and not supported in Copilot CLI (see .github/instructions/agents.instructions.md:18-19). This makes the Phase 4 instructions non-actionable in the intended environment. Remove model: from the invocation guidance, or document a Copilot-CLI-supported way to select models (if one exists) without relying on VS Code frontmatter fields.

Copilot uses AI. Check for mistakes.

**For each model**, invoke the try-fix skill:
```
state_file: CustomAgentLogsTmp/PRState/pr-XXXXX.md
Invoke the try-fix skill for PR #XXXXX:
- Platform: [android/ios]
- TestFilter: "IssueXXXXX"

Copilot AI Jan 31, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The suggested try-fix invocation prompt only provides Platform/TestFilter/state_file, but the try-fix skill documents required inputs as problem, test_command, target_files, and platform (see .github/skills/try-fix/references/example-invocation.md). Update the invocation template to include those required fields (and derive test_command from Platform/TestFilter if you want to keep TestFilter as a convenience).

Suggested change
- Platform: [android/ios]
- TestFilter: "IssueXXXXX"
- problem: Short description of the bug and expected behavior for PR #XXXXX (based on the linked issue and PR discussion)
- platform: [android/ios]
- test_filter: "IssueXXXXX" # Convenience filter name used to scope tests
- test_command: [INSERT test command here using the test_filter above, e.g., `dotnet test <TestProject>.csproj --filter "IssueXXXXX"`]
- target_files:
- [EDIT ME] src/<area>/<likely-affected-file-1>.cs
- [EDIT ME] src/<area>/<likely-affected-file-2>.cs

Copilot uses AI. Check for mistakes.
- state_file: CustomAgentLogsTmp/PRState/pr-XXXXX.md

Generate ONE independent fix idea and test it empirically.
Do NOT look at the PR's fix - generate ideas independently.

Copilot AI Jan 31, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The prompt says “Do NOT look at the PR's fix”, but the try-fix skill requires reviewing existing PR changes first to ensure the attempted approach is different (see .github/skills/try-fix/SKILL.md “Core Principles”, item 3, and Step 1). Consider rephrasing to: generate an initial idea independently, then review the PR’s fix only to avoid duplicating it.

Copilot uses AI. Check for mistakes.
```

try-fix will automatically append rows to the Fix Candidates table and set the "Exhausted" field. You remain responsible for:
- Setting "Selected Fix" field with reasoning
- Updating phase status to ✅ COMPLETE
**Wait for each to complete before starting the next.**

#### Round 2+: Cross-Pollination Loop

After Round 1 completes, share ALL results with ALL 5 models and ask for NEW ideas:

```
┌─────────────────────────────────────────────────────────────┐
Agent orchestration loop
Cross-Pollination Loop
├─────────────────────────────────────────────────────────────┤
│ │
│ attempts = 0 │
│ max_attempts = 5 │
│ state_file = "CustomAgentLogsTmp/PRState/pr-XXXXX.md" │
│ LOOP until no new ideas: │
│ │
│ while (attempts < max_attempts): │
│ result = invoke try-fix skill (with state_file) │
│ attempts++ │
│ 1. Compile summary of ALL try-fix attempts so far: │
│ - Approach tried │
│ - Pass/Fail result │
│ - Key learnings (why it worked or failed) │
│ │

Copilot AI Jan 31, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cross-pollination Step 1 requires compiling a summary of all try-fix attempts “so far” each loop iteration. This can quickly exceed Copilot CLI prompt limits (30,000 chars per .github/instructions/agents.instructions.md:27) and make the loop impractical. Add guidance to keep the shared summary bounded (e.g., fixed-size bullets per attempt) and/or add a maximum number of cross-pollination rounds/attempts.

Copilot uses AI. Check for mistakes.
│ if result.exhausted: │
│ break # try-fix has no more ideas │
│ 2. Share summary with ALL 5 models, ask each: │
│ "Given these results, do you have any NEW fix ideas │
│ that haven't been tried? If yes, describe briefly." │
│ │
# result.passed indicates if this attempt worked
# try-fix already recorded to state file
# Continue loop to explore more alternatives
3. For each model with a new idea:
→ Run try-fix with that model (SEQUENTIAL)
→ Wait for completion before next
│ │
# After loop: compare all try-fix results vs PR's fix
# Update "Exhausted" and "Selected Fix" fields
4. If ANY new ideas were tested → repeat loop
If NO new ideas from ANY model → exit loop
│ │
└─────────────────────────────────────────────────────────────┘
```

**Stop the loop when:**
- `try-fix` returns `exhausted=true` (no more ideas)
- 5 try-fix attempts have been made
- User requests to stop
**Exhaustion criteria**: The loop exits when ALL 5 models confirm they have no new ideas to try.

Copilot AI Jan 31, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new exhaustion rule (“ALL 5 models confirm no new ideas”) conflicts with how try-fix defines and updates the state file’s Exhausted field (it’s set per-invocation based on that run’s judgment; see .github/skills/try-fix/SKILL.md Step 9-10). Either align Phase 4 to the existing per-invocation exhaustion semantics, or introduce a separate multi-model exhaustion tracker so the state file doesn’t get contradictory updates.

Suggested change
**Exhaustion criteria**: The loop exits when ALL 5 models confirm they have no new ideas to try.
**Coordination loop stop condition**: Exit when a full round completes and NO model proposes any new fix ideas. This is separate from the per-invocation `Exhausted` flag that each `try-fix` run sets in the state file.

Copilot uses AI. Check for mistakes.

#### try-fix Invocation Details

Each `try-fix` invocation (via task agent):
- Reads state file to learn from prior attempts
- Reverts PR's fix to get broken baseline
- Proposes and implements ONE fix idea
- Runs tests to validate
- Records result with failure analysis
- Reverts changes (restores PR's fix)
- Updates state file with attempt results

See `.github/skills/try-fix/SKILL.md` for full details.

### What try-fix Does (Each Invocation)

Each `try-fix` invocation:
Each `try-fix` invocation (run via task agent with specific model):
1. Reads state file to learn from prior failed attempts
2. Reverts PR's fix to get a broken baseline
3. Proposes ONE new independent fix idea
4. Implements and tests it
5. Records result (with failure analysis if it failed)
6. **Updates state file** (appends row to Fix Candidates table if state_file provided)
6. **Updates state file** (appends row to Fix Candidates table)
7. Reverts all changes (restores PR's fix)
8. Returns `{passed: bool, exhausted: bool}`

See `.github/skills/try-fix/SKILL.md` for full details.

Expand Down Expand Up @@ -143,6 +174,7 @@ Update the state file:
- **try-fix found a simpler/better alternative** → Request changes with suggestion
- **try-fix found same solution independently** → Strong validation, approve PR
- **All try-fix attempts failed** → PR's fix is the only working solution, approve PR
- **Multiple passing alternatives** → Select simplest/most robust

### Step 4: Apply Selected Fix (if different from PR)

Expand All @@ -165,10 +197,11 @@ Update the state file:
5. Change 📋 Report status to `▶️ IN PROGRESS`

**Before marking ✅ COMPLETE, verify state file contains:**
- [ ] Root Cause Analysis filled in (if applicable)
- [ ] Round 1 completed: All 5 models ran try-fix
- [ ] Cross-pollination completed: All 5 models confirmed "no new ideas"
- [ ] Fix Candidates table has numbered rows for each try-fix attempt
- [ ] Each row has: approach, test result, files changed, notes
- [ ] "Exhausted" field set (Yes/No)
- [ ] "Exhausted" field set to Yes (all models confirmed no new ideas)
- [ ] "Selected Fix" populated with reasoning

Copilot AI Jan 31, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Phase 4 completion checklist no longer calls out Root Cause Analysis being filled in, but other repo guidance/templates expect root-cause documentation (e.g., pr-finalize template includes a “### Root Cause” section in .github/skills/pr-finalize/SKILL.md:199+). Consider adding an explicit checklist item indicating where the root cause must be documented (Fix section vs Report section) to avoid leaving it out.

Suggested change
- [ ] "Selected Fix" populated with reasoning
- [ ] "Selected Fix" populated with reasoning
- [ ] Root cause analysis documented for the selected fix (to be surfaced in 📋 Report phase “### Root Cause” section)

Copilot uses AI. Check for mistakes.
- [ ] No ⏳ PENDING markers remain in Fix section
- [ ] State file committed
Expand Down Expand Up @@ -287,8 +320,11 @@ Update all phase statuses to complete.

- ❌ **Looking at PR's fix before generating ideas** - Generate fix ideas independently first
- ❌ **Re-testing the PR's fix in try-fix** - Gate already validated it; try-fix tests YOUR ideas
- ❌ **Skipping the try-fix loop** - Always explore at least one independent alternative
- ❌ **Skipping models in Round 1** - All 5 models must run try-fix before cross-pollination
- ❌ **Running try-fix in parallel** - SEQUENTIAL ONLY - they modify same files and use same device
- ❌ **Stopping before cross-pollination** - Must share results and check for new ideas
- ❌ **Not analyzing why fixes failed** - Record the flawed reasoning to help future attempts
- ❌ **Selecting a failing fix** - Only select from passing candidates
- ❌ **Forgetting to revert between attempts** - Each try-fix must start from broken baseline, end with PR restored
- ❌ **Declaring exhaustion prematurely** - All 5 models must confirm "no new ideas"
- ❌ **Rushing the report** - Take time to write clear justification