Use PAT pool for agentic workflows - #36204
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 36204Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 36204" |
github-actions
Bot
added
the
area-infrastructure
There was a problem hiding this comment.
Pull request overview
This PR updates MAUI’s GitHub Agentic Workflows (gh-aw) to use a shared “Copilot PAT pool” mechanism, distributing Copilot CLI authentication across a set of environment-scoped PAT secrets to reduce single-token throttling and centralize onboarding.
Changes:
- Adds a reusable
shared/pat_pool.mdimport (plus onboarding README) to select a randomCOPILOT_PAT_{0-9}and wire it intoCOPILOT_GITHUB_TOKEN. - Migrates multiple gh-aw source workflows from the
gh-aw-agentsenvironment to the isolatedcopilot-pat-poolenvironment and overrides the Copilot engine token viacase(...). - Adds a scheduled/manual
validate-pat-pool.ymlworkflow and regenerates gh-aw lock files / maintenance workflow with gh-aw v0.80.9.
Reviewed changes
Copilot reviewed 23 out of 23 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/validate-pat-pool.yml | New workflow to validate each PAT slot and fail when the pool is invalid/unknown/empty. |
| .github/workflows/shared/pat_pool.md | New shared import that selects a PAT slot number and exposes it as pat_number. |
| .github/workflows/shared/pat_pool.README.md | Onboarding and usage documentation for importing and configuring the PAT pool. |
| .github/workflows/rerun-review-scanner.md | Migrates to copilot-pat-pool and overrides COPILOT_GITHUB_TOKEN using the PAT pool output. |
| .github/workflows/daily-repo-status.md | Migrates to copilot-pat-pool and overrides COPILOT_GITHUB_TOKEN using the PAT pool output. |
| .github/workflows/copilot-review-tests.md | Migrates to copilot-pat-pool and overrides COPILOT_GITHUB_TOKEN using the PAT pool output. |
| .github/workflows/copilot-evaluate-tests.md | Migrates to copilot-pat-pool and overrides COPILOT_GITHUB_TOKEN using the PAT pool output. |
| .github/workflows/ci-status-main.md | Migrates to copilot-pat-pool and overrides COPILOT_GITHUB_TOKEN using the PAT pool output. |
| .github/workflows/ci-status-net11.md | Migrates to copilot-pat-pool and overrides COPILOT_GITHUB_TOKEN using the PAT pool output. |
| .github/workflows/ci-status-fix.md | Migrates to copilot-pat-pool and overrides COPILOT_GITHUB_TOKEN using the PAT pool output. |
| .github/workflows/ci-status-fix-net11.md | Migrates to copilot-pat-pool and overrides COPILOT_GITHUB_TOKEN using the PAT pool output. |
| .github/workflows/agentic-labeler.md | Migrates to copilot-pat-pool and overrides COPILOT_GITHUB_TOKEN using the PAT pool output. |
| .github/workflows/agentics-maintenance.yml | Regenerated maintenance workflow with updated gh-aw version and behavior. |
| .github/workflows/copilot-evaluate-tests.lock.yml | Regenerated lock workflow reflecting pat_pool import, environment change, and new pins. |
| .github/workflows/agentic-labeler.lock.yml | Regenerated lock workflow reflecting pat_pool import, environment change, and new pins. |
| .github/workflows/daily-repo-status.lock.yml | Regenerated lock workflow reflecting pat_pool import, environment change, and new pins. |
| .github/workflows/rerun-review-scanner.lock.yml | Regenerated lock workflow reflecting pat_pool import, environment change, and new pins. |
| .github/workflows/copilot-review-tests.lock.yml | Regenerated lock workflow reflecting pat_pool import, environment change, and new pins. |
| .github/workflows/ci-status-main.lock.yml | Regenerated lock workflow reflecting pat_pool import, environment change, and new pins. |
| .github/workflows/ci-status-net11.lock.yml | Regenerated lock workflow reflecting pat_pool import, environment change, and new pins. |
| .github/workflows/ci-status-fix.lock.yml | Regenerated lock workflow reflecting pat_pool import, environment change, and new pins. |
| .github/workflows/ci-status-fix-net11.lock.yml | Regenerated lock workflow reflecting pat_pool import, environment change, and new pins. |
| .github/aw/actions-lock.json | Updates the gh-aw actions lock mapping for v0.80.9 and adds setup-cli. |
Copilot review noted the empty-pool step summary said agentic workflows 'will fall back to the default COPILOT_GITHUB_TOKEN', which read as benign even though this validation workflow exits 1 on an empty pool. Reword to state both facts: the check fails (to surface the gap), and the agentic workflows do fall back to the shared default until the pool is populated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@reviewers — addressed the one Copilot finding in |
MauiBot
left a comment
MauiBot
left a comment
There was a problem hiding this comment.
Expert Review — 2 findings
See inline comments for details.
|
|
||
| if [ $invalid -eq 0 ] && [ $unknown -eq 0 ] && [ $valid -eq 0 ]; then | ||
| echo "> [!WARNING]" | ||
| echo "> **Empty PAT pool** -- no PATs are configured, so this check fails (exit 1) to surface the gap. Agentic workflows themselves fall back to the shared default \`COPILOT_GITHUB_TOKEN\` (more throttling-prone), but the pool should be populated." |
There was a problem hiding this comment.
🔍 AI-Generated Review (multi-model)
[moderate] gh-aw PAT pool / token exposure messaging — This step-summary text (already tweaked once in this PR after review feedback) still asserts that "Agentic workflows themselves fall back to the shared default COPILOT_GITHUB_TOKEN" when the pool is empty. That fallback does not exist in the code shipped in this PR: grep -rn "secrets.COPILOT_GITHUB_TOKEN" .github/workflows/*.lock.yml .github/workflows/*.md .github/workflows/shared/*.md returns zero matches. Every consuming workflow's case(...) expression (see shared/pat_pool.md and all 7 *.md files importing it) ends with the literal string 'NO COPILOT PAT AVAILABLE' as the default, not secrets.COPILOT_GITHUB_TOKEN — so an empty pool causes the Copilot CLI to receive a bogus literal token (auth failure) rather than silently degrading to a shared token. On-call responders reading this summary will misdiagnose the failure mode. Recommend rewording to state the actual behavior (auth failure with a placeholder token) or implementing the described fallback in pat_pool.md's case() default value.
| done | ||
|
|
||
| # If none of the entries in the pool have values, emit a warning | ||
| # and do not set an output value. The consumer can fall back to |
There was a problem hiding this comment.
🔍 AI-Generated Review (multi-model)
[minor] Logic and Correctness (stale comment) — This comment ("The consumer can fall back to using COPILOT_GITHUB_TOKEN") describes a fallback that no consuming workflow implements: all 7 .md files that import this file terminate their case() expression with the literal 'NO COPILOT PAT AVAILABLE', never secrets.COPILOT_GITHUB_TOKEN. This is the same inaccurate claim flagged (and only partially addressed, in validate-pat-pool.yml) elsewhere in this PR. Either update this comment to reflect the actual fail-closed behavior, or add the fallback to the shared case() template in pat_pool.README.md so the comment becomes true.
This comment has been minimized.
This comment has been minimized.
MauiBot
added
s/agent-fix-pr-picked
MauiBot
left a comment
MauiBot
left a comment
There was a problem hiding this comment.
AI Review Summary
@kubaflo — new AI review results are available based on this last commit:
0f9faca. To request a fresh review after new comments or commits, comment/review rerun.
🗂️ Review Sessions — click to expand
🚦 Gate — Test Before & After Fix
Gate Result: ⚠️ SKIPPED
Gate skipped (SkipUITests fast test mode). No UI/device tests were run for this pipeline invocation.
📋 Pre-Flight — Context & Validation
Issue: N/A - No linked issue detected
PR: #36204 - Use PAT pool for agentic workflows
Platforms Affected: infrastructure / GitHub Actions workflows (testing platform requested: android, but no Android runtime code is affected)
Files Changed: 23 implementation/infrastructure, 0 test
Key Findings
- PR migrates agentic workflows to a shared
copilot-pat-poolenvironment and PAT selector, adds shared PAT-pool documentation/import files, addsvalidate-pat-pool.yml, and regenerates gh-aw lock files. - Gate result supplied by caller: SKIPPED because no tests were detected in this PR; do not re-run gate verification.
- GitHub CLI is unauthenticated in this environment, so PR metadata/comments were gathered via public GitHub API where possible; required-check state remains undetermined.
- Prior Copilot inline finding on
.github/workflows/validate-pat-pool.yml:160reported misleading empty-pool wording; author replied that wording was adjusted. - Independent code review still found a mismatch: docs/summary claim empty pool fallback to
COPILOT_GITHUB_TOKEN, but generated consumers use sentinelNO COPILOT PAT AVAILABLE.
Code Review Summary
Verdict: NEEDS_DISCUSSION
Confidence: low
Errors: 0 | Warnings: 1 | Suggestions: 0
Key code review findings:
- ⚠
.github/workflows/validate-pat-pool.yml:158and.github/workflows/shared/pat_pool.md:41-43still describe empty pools as falling back toCOPILOT_GITHUB_TOKEN, while generated workflows fail auth withNO COPILOT PAT AVAILABLE.
Fix Candidates
| # | Source | Approach | Test Result | Files Changed | Notes |
|---|---|---|---|---|---|
| PR | PR #36204 | Add shared PAT-pool import, validation workflow, docs, and regenerated gh-aw lock files | 23 files | Original PR |
🔬 Code Review — Deep Analysis
Code Review — PR #36204
Independent Assessment
What this changes: Migrates gh-aw workflows to a shared Copilot PAT pool, adds PAT-pool docs/import, validation workflow, and regenerated lock files.
Inferred motivation: Reduce throttling from one shared Copilot token and isolate token access through copilot-pat-pool.
Reconciliation with PR Narrative
Author claims: PR body is empty. Later comment says the empty-pool wording was clarified.
Agreement/disagreement: The code still says empty pools fall back to COPILOT_GITHUB_TOKEN, but workflows default to literal NO COPILOT PAT AVAILABLE.
Prior Review Reconciliation
| Prior ❌ Error Finding | Source | Status | Evidence |
|---|---|---|---|
| No prior ❌ Error findings found. | — | — | Checked review bodies, inline comments, and issue comments via public API. |
Blast Radius Assessment
- Runs for all instances: yes — all migrated agentic workflows use the new PAT-pool dependency.
- Startup impact: yes — empty/misconfigured pool can prevent agentic jobs from authenticating.
- Static/shared state: no — workflow/job configuration only.
CI Status
- Required-check result: unavailable/undetermined
- Classification: undetermined
- Action taken:
gh pr checks --requiredunavailable because GitHub CLI is unauthenticated; public check-runs showBuild Analysisin progress andmaui-prneutral, but required-check state cannot be determined. Confidence capped low; no LGTM.
Findings
⚠️ Warning — Empty-pool fallback documentation is still inaccurate
.github/workflows/validate-pat-pool.yml:158 says empty pools fall back to the shared default COPILOT_GITHUB_TOKEN, and .github/workflows/shared/pat_pool.md:41-43 says consumers can fall back. The generated consumers use case(..., 'NO COPILOT PAT AVAILABLE'), so an empty pool fails authentication instead of falling back. This can mislead responders during PAT-pool outages.
Failure-Mode Probing
- Empty PAT pool: validation fails, and agentic workflows receive
NO COPILOT PAT AVAILABLE; no fallback occurs. - Invalid PAT: validation fails; selected invalid PAT would cause workflow auth failure.
- Fork/
pull_request_target: workflow config is restored/generated and token selection occurs in environment-scoped jobs, so no direct PR-controlled token exposure found.
Verdict: NEEDS_DISCUSSION
Confidence: low
Summary: The implementation generally follows the intended PAT-pool design, but stale fallback wording remains. CI required-check status is undetermined due unauthenticated gh and pending public checks, so the skill rules prohibit LGTM.
🛠️ Fix — Analysis & Comparison
Fix Candidates
| # | Source | Approach | Test Result | Files Changed | Notes |
|---|---|---|---|---|---|
| 1 | maui-expert-reviewer + code-review loop | Correct empty PAT-pool docs/validation to describe fail-fast sentinel behavior instead of shared-token fallback | ✅ PASS | 2 files | Fixes the remaining code-review warning while preserving PR behavior and token isolation |
| PR | PR #36204 | Add shared PAT-pool import, validation workflow, docs, and regenerated gh-aw lock files | 23 files | Original PR; no tests detected by prior gate |
Cross-Pollination
| Model/Reviewer | Round | New Ideas? | Details |
|---|---|---|---|
| maui-expert-reviewer | 1 | Yes | Proposed fail-fast empty-pool documentation/validation alignment. |
Exhausted: No — stopped because Candidate #1 passed all available validation and is demonstrably better than the PR's current fix for the identified mismatch.
Selected Fix: Candidate #1 — it resolves the stale fallback wording without weakening the PAT-pool security design or changing generated workflow behavior.
📝 Recommended PR Title & Description
Assessment: ✏️ Recommend updating — the current title is understandable but not component-scoped, and the description was unavailable/null so it does not capture the PAT-pool migration, validation workflow, regenerated gh-aw locks, or the fail-fast empty-pool behavior.
Recommended title
[CI] Agentic workflows: Use Copilot PAT pool
Recommended description
### Description of Change
Migrates the gh-aw agentic workflows to use a shared `copilot-pat-pool` environment instead of relying on one shared Copilot token.
Key changes:
- Adds `.github/workflows/shared/pat_pool.md` to select a configured `COPILOT_PAT_0` through `COPILOT_PAT_9` entry and expose the selected pool slot to consuming workflows.
- Adds `.github/workflows/shared/pat_pool.README.md` with onboarding and usage guidance for the PAT pool import.
- Updates the agentic workflow source files to import the PAT pool and map `COPILOT_GITHUB_TOKEN` from the selected pool entry.
- Adds `.github/workflows/validate-pat-pool.yml` to validate configured PAT pool entries and surface invalid, unknown, or empty-pool states.
- Regenerates the affected gh-aw lock files and updates gh-aw action locks for v0.80.9.
When no PAT pool entry is configured, the validation workflow fails to surface the gap and the agentic workflows fail fast with `NO COPILOT PAT AVAILABLE` until the pool is populated.
### Issues Fixed
None.
🏁 Report — Final Recommendation
Comparative Analysis — PR #36204
Candidates compared
| Rank | Candidate | Regression/test status | Assessment |
|---|---|---|---|
| 1 | pr-plus-reviewer |
PASS — targeted static validation found no stale fallback wording and confirmed fail-fast sentinel documentation | Best candidate. It keeps the PR's PAT-pool implementation, applies the expert review feedback, and aligns operational docs/validation output with the generated workflow behavior. |
| 2 | try-fix-1 |
PASS — same targeted static validation passed | Functionally equivalent to pr-plus-reviewer; ranked second only because the reviewer-applied PR candidate is the direct evolution of the submitted PR fix. |
| 3 | pr |
SKIPPED gate; expert review found 2 actionable wording mismatches | The raw PR implements the PAT-pool migration, but still claims empty pools fall back to COPILOT_GITHUB_TOKEN while generated consumers actually use NO COPILOT PAT AVAILABLE. |
Key comparison
The core implementation in the raw PR is sound: it introduces environment-scoped PAT selection, migrates the gh-aw workflows, adds validation, and regenerates lock files. The remaining problem is documentation/diagnostic accuracy for the empty-pool failure mode. Because the generated COPILOT_GITHUB_TOKEN mappings end with NO COPILOT PAT AVAILABLE, an empty pool fails authentication rather than falling back to the prior shared token.
Both pr-plus-reviewer and try-fix-1 fix that mismatch without weakening token isolation or changing generated workflow behavior. The two candidates are effectively the same patch; pr-plus-reviewer wins because it represents the PR fix plus the expert reviewer's actionable feedback applied in the requested sandbox candidate.
Winner
Winner: pr-plus-reviewer
Rationale: It preserves the PR's intended PAT-pool design, fixes the only actionable expert-review issue, and passed the available targeted validation. The raw PR ranks lower because it retains misleading fallback wording, and no candidate with failed regression tests outranks a passing candidate.
🧭 Next Steps — review latest findings
No alternative fix was selected for this run. Review the session findings and CI results before merging.
The evaluate and hermeticity-gate jobs read a Copilot token from secrets, but the token lived only in a GitHub environment those jobs never declared, so the secret resolved to an empty string and the 'Select Copilot token' step failed with 'No token configured'. The gh-aw fleet migrated to the copilot-pat-pool environment (COPILOT_PAT_0..9) in #36204 and the old gh-aw-agents environment / COPILOT_GITHUB_TOKEN secret were deleted. This hand-authored workflow was the only token consumer not migrated. Fix: declare 'environment: copilot-pat-pool' on both jobs and repoint the token-selection steps to randomly pick a populated COPILOT_PAT_* secret. The env var name passed to the Copilot CLI is unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…on PAT-pool main Enhance the CI Failure Fixer workflows (main + net11.0) so that, instead of opening one draft [ci-fix] PR per ci-scan issue and handing off to a human forever, the workflow WATCHES its own open PRs: when the fix's own CI settles red and the red is caused by the fix itself, it pushes a fresh follow-up commit onto the SAME PR branch — up to 10 attempts — then defers to humans. Greens are surfaced for review; unrelated flake is annotated without burning an attempt; human engagement on a PR always yields to the human. Key pieces: - New .github/scripts/Query-CiFixPRs.ps1: deterministic pre-agent prefetch that writes each open [ci-fix] PR's head-SHA-matched CI state, human-engagement flag, and ci-fix-attempts marker to JSON the agent consumes (no blind-query). - Both twins: watch state machine (surface-green / annotate-flake / advance), attempt counter via <!-- ci-fix-attempts: N/10 --> body marker, safe-outputs for push-to-pull-request-branch + add-comment + update-pull-request + add-labels (allow-listed), and a pre-activation prefetch job scoped with the CI-read permissions the metadata queries need. - Track C: respond to write-access maintainer change-requests (author-gated against bots, per-review RID dedup) with fully-paginated review + inline-comment reads. - AIC budget disabled (-1) for these workflows; abuse control rests on the safe-output caps, the 10-attempt ceiling, allowed-files, and title/label/branch gates. Rebased onto main after #36204 (PAT pool for agentic workflows): the loop now coexists with the copilot-pat-pool environment + COPILOT_GITHUB_TOKEN PAT selector, with the pre-activation prefetch keeping its real CI-read scopes. Compiled with gh-aw v0.80.9 to match the repo baseline; actions-lock.json untouched. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
No description provided.