Improve evaluate-tests workflow and remove obsolete files - #35259
Conversation
- Remove Checkout-GhAwPr.ps1 (116 lines) — checkout not needed, agent reads PR data via MCP tools - Remove gh-aw-workflows.instructions.md (replaced by gh-aw-guide skill) - Simplify copilot-evaluate-tests.md: - Remove checkout step (agent uses MCP tools, not filesystem) - Add roles: [admin, maintain, write] (was missing) - Fix cancel-in-progress: true → false (slash_command best practice) - Update prompt to reference MCP tools instead of local files Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 35259Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 35259" |
There was a problem hiding this comment.
Pull request overview
Updates the copilot-evaluate-tests gh-aw workflow configuration/prompting and removes two .github/ files intended to be obsolete, with the goal of relying more on MCP-based PR access and tightening workflow activation security.
Changes:
- Add
roles: [admin, maintain, write]gating and setcancel-in-progress: falsein the evaluate-tests workflow. - Remove the workflow_dispatch-only “Checkout PR and restore agent infrastructure” step from
copilot-evaluate-tests.md. - Delete
.github/scripts/Checkout-GhAwPr.ps1and.github/instructions/gh-aw-workflows.instructions.md.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| .github/workflows/copilot-evaluate-tests.md | Adds role gating, adjusts concurrency cancellation, and updates the agent prompt to emphasize MCP-based PR access while removing a checkout/restore step. |
| .github/workflows/copilot-evaluate-tests.lock.yml | Regenerates the compiled workflow (gh-aw v0.68.3), reflecting the source workflow changes and updated gh-aw runtime scaffolding. |
| .github/scripts/Checkout-GhAwPr.ps1 | Removes the shared checkout + trusted-infra restore script (used by other workflows). |
| .github/instructions/gh-aw-workflows.instructions.md | Removes the gh-aw authoring guidance instruction file. |
| fi | ||
| echo "✅ Found test files to evaluate:" | ||
| echo "$TEST_FILES" | head -20 | ||
|
|
||
| # For slash_command triggers, the gh-aw platform's checkout_pr_branch.cjs runs | ||
| # AFTER all user steps and overlays the PR branch onto the workspace. This means | ||
| # fork PRs can supply their own .github/skills/ and .github/instructions/. | ||
| # We cannot restore trusted infra here because the platform checkout runs later. | ||
| # Mitigation: agent is sandboxed (no credentials), max 1 comment via safe-outputs, | ||
| # and the agent prompt includes a pre-flight check that catches missing SKILL.md. | ||
| # See: .github/instructions/gh-aw-workflows.instructions.md "The issue_comment + Fork Problem" | ||
|
|
||
| # For workflow_dispatch, the platform skips checkout entirely — this step is the | ||
| # only thing that gets the PR code onto disk and restores trusted infra from main. | ||
| - name: Checkout PR and restore agent infrastructure | ||
| if: github.event_name == 'workflow_dispatch' | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| PR_NUMBER: ${{ inputs.pr_number }} | ||
| run: pwsh .github/scripts/Checkout-GhAwPr.ps1 | ||
| --- |
There was a problem hiding this comment.
This PR deletes .github/scripts/Checkout-GhAwPr.ps1, but that script is still referenced by other gh-aw workflows (e.g., .github/workflows/shared/review-shared.md and the generated review.agent.lock.yml). Removing it will break those workflows the next time they run (notably on workflow_dispatch). Either keep this script, or update all remaining references (and their behavior) in the same PR before deleting it.
| - **PR Number**: ${{ github.event.issue.number || inputs.pr_number }} | ||
|
|
||
| The PR branch has been checked out for you. All files from the PR are available locally. | ||
| The PR is available via MCP tools. Use `gh pr view` and `gh pr diff` to access PR data. |
There was a problem hiding this comment.
The prompt says the PR is available via MCP tools but then tells the agent to use gh pr view/gh pr diff. In this workflow, the agent run generally doesn’t have a GH_TOKEN env var, so gh commands (and the Gather-TestContext.ps1 -PrNumber ... path) may fail or be inconsistent across triggers. Suggest updating the instructions to primarily use GitHub MCP tools for PR metadata/diff (and, if needed, move Gather-TestContext.ps1 into pre-agent steps: where GH_TOKEN is available, then have the agent read the generated report).
| The PR is available via MCP tools. Use `gh pr view` and `gh pr diff` to access PR data. | |
| The PR is available via GitHub MCP tools. Use MCP tools to inspect PR metadata, changed files, and diffs from inside the agent. Do not rely on `gh pr view` or `gh pr diff` in the agent environment; those are for pre-agent `steps:` only, where `GH_TOKEN` is available. |
Note
Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!
Description
Simplifies the evaluate-tests gh-aw workflow and removes obsolete files.
Changes
Workflow improvements (
copilot-evaluate-tests.md):roles: [admin, maintain, write](was missing — security gap)cancel-in-progress: true→false(slash_command best practice per gh-aw guide)Removed files:
Checkout-GhAwPr.ps1(116 lines) — no longer needed since workflows use MCP toolsgh-aw-workflows.instructions.md(290 lines) — replaced by the gh-aw-guide skill