diff --git a/.github/skills/ci-analysis/SKILL.md b/.github/skills/ci-analysis/SKILL.md index 5b42f8363ee1b2..17be7cd924d36a 100644 --- a/.github/skills/ci-analysis/SKILL.md +++ b/.github/skills/ci-analysis/SKILL.md @@ -1,6 +1,13 @@ --- name: ci-analysis -description: Analyze CI build and test status from Azure DevOps and Helix for dotnet repository PRs. Use when checking CI status, investigating failures, determining if a PR is ready to merge, or given URLs containing dev.azure.com or helix.dot.net. Also use when asked "why is CI red", "test failures", "retry CI", "rerun tests", "is CI green", "build failed", "checks failing", or "flaky tests". +description: > + Analyze CI build and test status from Azure DevOps and Helix for dotnet repository PRs. + Use when checking CI status, investigating failures, determining if a PR is ready to merge, + or given URLs containing dev.azure.com or helix.dot.net. Also use when asked "why is CI red", + "test failures", "retry CI", "rerun tests", "is CI green", "build failed", "checks failing", + or "flaky tests". DO NOT USE FOR: investigating stale codeflow PRs or dependency update health, + tracing whether a commit has flowed from one repo to another, reviewing code changes for + correctness or style. --- # Azure DevOps and Helix CI Analysis @@ -9,27 +16,20 @@ Analyze CI build status and test failures in Azure DevOps and Helix for dotnet r > 🚨 **NEVER** use `gh pr review --approve` or `--request-changes`. Only `--comment` is allowed. Approval and blocking are human-only actions. -**Workflow**: Gather PR context (Step 0) β†’ run the script β†’ read the human-readable output + `[CI_ANALYSIS_SUMMARY]` JSON β†’ synthesize recommendations yourself. The script collects data; you generate the advice. For supplementary investigation beyond the script, MCP tools (AzDO, Helix, GitHub) provide structured access when available; the script and `gh` CLI work independently when they're not. +**Workflow**: Gather PR context (Step 0) β†’ run the script β†’ read human-readable output + `[CI_ANALYSIS_SUMMARY]` JSON β†’ synthesize recommendations. The script collects data; you generate the advice. MCP tools (AzDO, Helix, GitHub) provide supplementary access when available; the script and `gh` CLI work independently when they're not. + +**Accessing services**: There are several possible methods to access each service (AzDO, Helix, GitHub). Start with MCP tools, then fall back to CLI (`gh` for GitHub, `Invoke-RestMethod` for AzDO/Helix REST APIs). Explore all available options before determining you don't have access. For AzDO, multiple tool sets may exist for different organizations β€” match the org in the build URL to the correct tools (see [references/azdo-helix-reference.md](references/azdo-helix-reference.md#azure-devops-organizations)). If queries return null, check the org before trying other approaches. For complex investigations, track what you've tried in SQL to avoid repeating failed approaches. ## When to Use This Skill -Use this skill when: -- Checking CI status on a PR ("is CI passing?", "what's the build status?", "why is CI red?") -- Investigating CI failures or checking why a PR's tests are failing -- Determining if a PR is ready to merge based on CI results -- Debugging Helix test issues or analyzing build errors -- Given URLs containing `dev.azure.com`, `helix.dot.net`, or GitHub PR links with failing checks -- Asked questions like "why is this PR failing", "analyze the CI", "is CI green", "retry CI", "rerun tests", or "test failures" +- Checking CI status ("is CI passing?", "why is CI red?") +- Investigating CI failures or determining merge readiness +- Debugging Helix test issues or build errors +- URLs containing `dev.azure.com`, `helix.dot.net`, or GitHub PR links with failing checks +- Questions like "retry CI", "rerun tests", "test failures", "checks failing" - Investigating canceled or timed-out jobs for recoverable results -## Script Limitations - -The `Get-CIStatus.ps1` script targets **Azure DevOps + Helix** infrastructure specifically. It won't help with: -- **GitHub Actions** workflows (different API, different log format) -- Repos not using **Helix** for test distribution (no Helix work items to query) -- Pure **build performance** questions (use MSBuild binlog analysis instead) - -However, the analysis patterns in this skill (interpreting failures, correlating with PR changes, distinguishing infrastructure vs. code issues) apply broadly even outside AzDO/Helix. +**Not for**: GitHub Actions workflows, non-Helix repos, or build performance (use binlog analysis). ## Quick Start @@ -45,215 +45,92 @@ However, the analysis patterns in this skill (interpreting failures, correlating # Other dotnet repositories ./scripts/Get-CIStatus.ps1 -PRNumber 12345 -Repository "dotnet/aspnetcore" -./scripts/Get-CIStatus.ps1 -PRNumber 67890 -Repository "dotnet/sdk" -./scripts/Get-CIStatus.ps1 -PRNumber 11111 -Repository "dotnet/roslyn" ``` -## Key Parameters - -| Parameter | Description | -|-----------|-------------| -| `-PRNumber` | GitHub PR number to analyze | -| `-BuildId` | Azure DevOps build ID | -| `-ShowLogs` | Fetch and display Helix console logs | -| `-Repository` | Target repo (default: dotnet/runtime) | -| `-MaxJobs` | Max failed jobs to show (default: 5) | -| `-SearchMihuBot` | Search MihuBot for related issues | - -## Three Modes - -The script operates in three distinct modes depending on what information you have: - -| You have... | Use | What you get | -|-------------|-----|-------------| -| A GitHub PR number | `-PRNumber 12345` | Full analysis: all builds, failures, known issues, structured JSON summary | -| An AzDO build ID | `-BuildId 1276327` | Single build analysis: timeline, failures, Helix results | -| A Helix job ID (optionally a specific work item) | `-HelixJob "..." [-WorkItem "..."]` | Deep dive: list work items for the job, or with `-WorkItem`, focus on a single work item's console logs, artifacts, and test results | - -> ❌ **Don't guess the mode.** If the user gives a PR URL, use `-PRNumber`. If they paste an AzDO build link, extract the build ID. If they reference a specific Helix job, use `-HelixJob`. - -## What the Script Does - -### PR Analysis Mode (`-PRNumber`) -1. Discovers AzDO builds associated with the PR (from GitHub check status; for full build history, query AzDO builds on `refs/pull/{PR}/merge` branch) -2. Fetches Build Analysis for known issues -3. Gets failed jobs from Azure DevOps timeline -4. **Separates canceled jobs from failed jobs** (canceled may be dependency-canceled or timeout-canceled) -5. Extracts Helix work item failures from each failed job -6. Fetches console logs (with `-ShowLogs`) -7. Searches for known issues with "Known Build Error" label -8. Correlates failures with PR file changes -9. **Emits structured summary** β€” `[CI_ANALYSIS_SUMMARY]` JSON block with all key facts for the agent to reason over - -> **After the script runs**, you (the agent) generate recommendations. The script collects data; you synthesize the advice. See [Generating Recommendations](#generating-recommendations) below. - -### Build ID Mode (`-BuildId`) -1. Fetches the build timeline directly (skips PR discovery) -2. Performs steps 3–7 from PR Analysis Mode, but does **not** fetch Build Analysis known issues or correlate failures with PR file changes (those require a PR number). Still emits `[CI_ANALYSIS_SUMMARY]` JSON. - -### Helix Job Mode (`-HelixJob` [and optional `-WorkItem`]) -1. With `-HelixJob` alone: enumerates work items for the job and summarizes their status -2. With `-HelixJob` and `-WorkItem`: queries the specific work item for status and artifacts -3. Fetches console logs and file listings, displays detailed failure information - -## Interpreting Results - -**Known Issues section**: Failures matching existing GitHub issues - these are tracked and being investigated. - -**Build Analysis check status**: The "Build Analysis" GitHub check is **green** only when *every* failure is matched to a known issue. If it's **red**, at least one failure is unaccounted for β€” do NOT claim "all failures are known issues" just because some known issues were found. You must verify each failing job is covered by a specific known issue before calling it safe to retry. - -**Canceled/timed-out jobs**: Jobs canceled due to earlier stage failures or AzDO timeouts. Dependency-canceled jobs don't need investigation. **Timeout-canceled jobs may have all-passing Helix results** β€” the "failure" is just the AzDO job wrapper timing out, not actual test failures. To verify: use `hlx_status` on each Helix job in the timed-out build (include passed work items). If all work items passed, the build effectively passed. - -> ❌ **Don't dismiss timed-out builds.** A build marked "failed" due to a 3-hour AzDO timeout can have 100% passing Helix work items. Check before concluding it failed. - -**PR Change Correlation**: Files changed by PR appearing in failures - likely PR-related. - -**Build errors**: Compilation failures need code fixes. - -**Helix failures**: Test failures on distributed infrastructure. +For full parameter reference and mode details, see [references/script-modes.md](references/script-modes.md). -**Local test failures**: Some repos (e.g., dotnet/sdk) run tests directly on build agents. These can also match known issues - search for the test name with the "Known Build Error" label. +## Step 0: Gather Context (before running anything) -**Per-failure details** (`failedJobDetails` in JSON): Each failed job includes `errorCategory`, `errorSnippet`, and `helixWorkItems`. Use these for per-job classification instead of applying a single `recommendationHint` to all failures. +Context changes how you interpret every failure. **Don't skip this.** -Error categories: `test-failure`, `build-error`, `test-timeout`, `crash` (exit codes 139/134/-4), `tests-passed-reporter-failed` (all tests passed but reporter crashed β€” genuinely infrastructure), `unclassified` (investigate manually). - -> ⚠️ **`crash` does NOT always mean tests failed.** Exit code -4 often means the Helix work item wrapper timed out *after* tests completed. Always check `testResults.xml` before concluding a crash is a real failure. See [Recovering Results from Crashed/Canceled Jobs](#recovering-results-from-crashedcanceled-jobs). - -> ⚠️ **Be cautious labeling failures as "infrastructure."** Only conclude infrastructure with strong evidence: Build Analysis match, identical failure on target branch, or confirmed outage. Exception: `tests-passed-reporter-failed` is genuinely infrastructure. - -> ❌ **Missing packages on flow PRs β‰  infrastructure.** Flow PRs can cause builds to request *different* packages. Check *which* package and *why* before assuming feed delay. - -### Recovering Results from Crashed/Canceled Jobs - -When an AzDO job is canceled (timeout) or Helix work items show `Crash` (exit code -4), the tests may have actually passed. Follow this procedure: - -1. **Find the Helix job IDs** β€” Read the AzDO "Send to Helix" step log and search for lines containing `Sent Helix Job`. Extract the job GUIDs. - -2. **Check Helix job status** β€” Get pass/fail summary for each job. Look at `failedCount` vs `passedCount`. - -3. **For work items marked Crash/Failed** β€” Check if tests actually passed despite the crash. Try structured test results first (TRX parsing), then search for pass/fail counts in result files without downloading, then download as last resort: - - Parse the XML: `total`, `passed`, `failed` attributes on the `` element - - If `failed=0` and `passed > 0`, the tests passed β€” the "crash" is the wrapper timing out after test completion - -4. **Verdict**: - - All work items passed or crash-with-passing-results β†’ **Tests effectively passed.** The failure is infrastructure (wrapper timeout). - - Some work items have `failed > 0` in testResults.xml β†’ **Real test failures.** Investigate those specific tests. - - No testResults.xml uploaded β†’ Tests may not have run at all. Check console logs for errors. - -> This pattern is common with long-running test suites (e.g., WasmBuildTests) where tests complete but the Helix work item wrapper exceeds its timeout during result upload or cleanup. - -## Generating Recommendations - -After the script outputs the `[CI_ANALYSIS_SUMMARY]` JSON block, **you** synthesize recommendations. Do not parrot the JSON β€” reason over it. - -### Decision logic +1. **Read PR metadata** β€” title, description, author, labels, linked issues +2. **Classify the PR type**: -Read `recommendationHint` as a starting point, then layer in context: +| PR Type | How to detect | Interpretation shift | +|---------|--------------|---------------------| +| **Code PR** | Human author, code changes | Failures likely relate to the changes | +| **Flow/Codeflow PR** | Author is `dotnet-maestro[bot]`, "Update dependencies" | Missing packages may be behavioral, not infrastructure | +| **Backport** | Title mentions "backport", targets release branch | Check if test exists on target branch | +| **Merge PR** | Merging between branches | Conflicts cause failures, not individual changes | +| **Dependency update** | Bumps package versions, global.json | Build failures often trace to the dependency | -| Hint | Action | -|------|--------| -| `BUILD_SUCCESSFUL` | No failures. Confirm CI is green. | -| `KNOWN_ISSUES_DETECTED` | Known tracked issues found β€” but this does NOT mean all failures are covered. Check the Build Analysis check status: if it's red, some failures are unmatched. Only recommend retry for failures that specifically match a known issue; investigate the rest. | -| `LIKELY_PR_RELATED` | Failures correlate with PR changes. Lead with "fix these before retrying" and list `correlatedFiles`. | -| `POSSIBLY_TRANSIENT` | Failures could not be automatically classified β€” does NOT mean they are transient. Use `failedJobDetails` to investigate each failure individually. | -| `REVIEW_REQUIRED` | Could not auto-determine cause. Review failures manually. | -| `MERGE_CONFLICTS` | PR has merge conflicts β€” CI won't run. Tell the user to resolve conflicts. Offer to analyze a previous build by ID. | -| `NO_BUILDS` | No AzDO builds found (CI not triggered). Offer to check if CI needs to be triggered or analyze a previous build. | +3. **Check existing comments** β€” has someone already diagnosed failures or is a retry pending? +4. **Note the changed files** β€” you'll use these for correlation after the script runs -Then layer in nuance the heuristic can't capture: +## After the Script: Use Its Output -- **Mixed signals**: Some failures match known issues AND some correlate with PR changes β†’ separate them. Known issues = safe to retry; correlated = fix first. -- **Canceled jobs with recoverable results**: If `canceledJobNames` is non-empty, mention that canceled jobs may have passing Helix results (see "Recovering Results from Crashed/Canceled Jobs"). -- **Build still in progress**: If `lastBuildJobSummary.pending > 0`, note that more failures may appear. -- **Multiple builds**: If `builds` has >1 entry, `lastBuildJobSummary` reflects only the last build β€” use `totalFailedJobs` for the aggregate count. -- **BuildId mode**: `knownIssues` and `prCorrelation` won't be populated. Say "Build Analysis and PR correlation not available in BuildId mode." +> 🚨 **The script already collected the data. Do NOT re-query AzDO or Helix for information the script already produced.** Parse the `[CI_ANALYSIS_SUMMARY]` JSON and the human-readable output first. Only make additional API calls for data the script *didn't* provide (e.g., deeper Helix log searches, binlog analysis, build progression). -### How to Retry +**If the script found no builds** (e.g., AzDO builds expired, CI not triggered): report this to the user immediately. Don't spend turns re-querying AzDO with different org/project combinations β€” if the script couldn't find builds, they're likely unavailable. Offer alternatives: analyze by build ID if the user has one, check GitHub PR status for summary info, or note that Helix results may still be queryable directly even when AzDO builds have expired. -- **AzDO builds**: Comment `/azp run {pipeline-name}` on the PR (e.g., `/azp run dotnet-sdk-public`) -- **All pipelines**: Comment `/azp run` to retry all failing pipelines -- **Helix work items**: Cannot be individually retried β€” must re-run the entire AzDO build +**If the script succeeded**: the `[CI_ANALYSIS_SUMMARY]` JSON contains `failedJobDetails`, `knownIssues`, `canceledJobNames`, `prCorrelation`, and `recommendationHint`. Use these fields β€” don't re-fetch the same data via MCP tools or REST APIs. To find specific details in large output, use `Select-String` or `grep` on the output file rather than re-running the script. -### Tone and output format +> 🚨 **Check build progression on multi-commit PRs.** If the PR has multiple commits, query AzDO for builds on `refs/pull/{PR}/merge` (sorted by queue time, top 10-20) β€” `gh pr checks` only shows the latest SHA. Present a progression table showing which builds passed/failed at which SHAs. This narrows failures to the commit that introduced them. See [references/build-progression-analysis.md](references/build-progression-analysis.md). -Be direct. Lead with the most important finding. Structure your response as: -1. **Summary verdict** (1-2 sentences) β€” Is CI green? Failures PR-related? Known issues? -2. **Failure details** (2-4 bullets) β€” what failed, why, evidence -3. **Recommended actions** (numbered) β€” retry, fix, investigate. Include `/azp run` commands. +Then follow the detailed workflow in [references/analysis-workflow.md](references/analysis-workflow.md). Key principles: -Synthesize from: JSON summary (structured facts) + human-readable output (details/logs) + Step 0 context (PR type, author intent). +1. **Cross-reference failures with known issues** β€” The script outputs `failedJobDetails` and `knownIssues` as separate lists. You must explicitly match each failure to a known issue (by error message, test name, or job type) or mark it **unmatched**. Don't present them as two independent lists β€” the user needs a per-failure verdict. +2. **Check Build Analysis status** β€” Green = all failures matched known issues. Red = some unmatched. Never claim "all known issues" when Build Analysis is red. +3. **Correlate with PR changes** β€” same files failing = likely PR-related. +4. **Verify before claiming** β€” don't call it "infrastructure" without Build Analysis match or target-branch verification. Don't call it "safe to retry" unless ALL failures are accounted for. -## Analysis Workflow +For interpreting error categories, crash recovery, and canceled jobs: [references/failure-interpretation.md](references/failure-interpretation.md) -### Step 0: Gather Context (before running anything) +For generating recommendations from `[CI_ANALYSIS_SUMMARY]` JSON: [references/recommendation-generation.md](references/recommendation-generation.md) -Before running the script, read the PR to understand what you're analyzing. Context changes how you interpret every failure. +## Presenting Results -1. **Read PR metadata** β€” title, description, author, labels, linked issues -2. **Classify the PR type** β€” this determines your interpretation framework: +> 🚨 **Keep tables narrow β€” 4 short columns max (# | Job | Verdict | Issue).** Put error descriptions, work item lists, and evidence in **detail bullets below the table**, not in cells. Wide tables wrap and become unreadable in terminals. -| PR Type | How to detect | Interpretation shift | -|---------|--------------|---------------------| -| **Code PR** | Human author, code changes | Failures likely relate to the changes | -| **Flow/Codeflow PR** | Author is `dotnet-maestro[bot]`, title mentions "Update dependencies" | Missing packages may be behavioral, not infrastructure (see anti-pattern below) | -| **Backport** | Title mentions "backport", targets a release branch | Failures may be branch-specific; check if test exists on target branch | -| **Merge PR** | Merging between branches (e.g., release β†’ main) | Conflicts and merge artifacts cause failures, not the individual changes | -| **Dependency update** | Bumps package versions, global.json changes | Build failures often trace to the dependency, not the PR's own code | +> 🚨 **Use markdown links** for PRs (`[#121195](url)`), builds (`[Build 1305302](url)`), and jobs (`[job name](azdo-job-url)`). The script output and MCP tools provide URLs β€” thread them through. -3. **Check existing comments** β€” has someone already diagnosed the failures? Is there a retry pending? -4. **Note the changed files** β€” you'll use these to evaluate correlation after the script runs +Lead with a 1-2 sentence verdict, then the summary table, then detail bullets (one per failure), then recommended actions. For the full format example: [references/recommendation-generation.md](references/recommendation-generation.md). -> ❌ **Don't skip Step 0.** Running the script without PR context leads to misdiagnosis β€” especially for flow PRs where "package not found" looks like infrastructure but is actually a code issue. +## Anti-Patterns -### Step 1: Run the script +> 🚨 **Every failure verdict needs evidence β€” no "Likely flaky" without proof.** Each row in your summary table must cite a specific source: known issue number, Build Analysis match, or target-branch verification. If Build Analysis didn't match it and you haven't verified the target branch, the verdict is **"Unmatched β€” needs investigation"**, not "Likely flaky." A test that *looks* like it could be flaky is not the same as one you've *verified* is flaky. -Run with `-ShowLogs` for detailed failure info. +> ❌ **Don't label failures "infrastructure" without evidence.** Requires: Build Analysis match, identical failure on target branch, or confirmed outage. Exception: `tests-passed-reporter-failed` is genuinely infrastructure. -### Step 2: Analyze results +> ❌ **Don't dismiss timed-out builds.** A build "failed" due to AzDO timeout can have 100% passing Helix work items. Check Helix job status before concluding failure. -1. **Check Build Analysis** β€” If the Build Analysis GitHub check is **green**, all failures matched known issues and it's safe to retry. If it's **red**, some failures are unaccounted for β€” you must identify which failing jobs are covered by known issues and which are not. For 3+ failures, use SQL tracking to avoid missed matches (see [references/sql-tracking.md](references/sql-tracking.md)). -2. **Correlate with PR changes** β€” Same files failing = likely PR-related -3. **Compare with baseline** β€” If a test passes on the target branch but fails on the PR, compare Helix binlogs. See [references/binlog-comparison.md](references/binlog-comparison.md) β€” **delegate binlog download/extraction to subagents** to avoid burning context on mechanical work. -4. **Check build progression** β€” If the PR has multiple builds (multiple pushes), check whether earlier builds passed. A failure that appeared after a specific push narrows the investigation to those commits. See [references/build-progression-analysis.md](references/build-progression-analysis.md). Present findings as facts, not fix recommendations. -5. **Interpret patterns** (but don't jump to conclusions): - - Same error across many jobs β†’ Real code issue - - Build Analysis flags a known issue β†’ That *specific failure* is safe to retry (but others may not be) - - Failure is **not** in Build Analysis β†’ Investigate further before assuming transient - - Device failures, Docker pulls, network timeouts β†’ *Could* be infrastructure, but verify against the target branch first - - Test timeout but tests passed β†’ Executor issue, not test failure -6. **Check for mismatch with user's question** β€” The script only reports builds for the current head SHA. If the user asks about a job, error, or cancellation that doesn't appear in the results, **ask** if they're referring to a prior build. Common triggers: - - User mentions a canceled job but `canceledJobNames` is empty - - User says "CI is failing" but the latest build is green - - User references a specific job name not in the current results - Offer to re-run with `-BuildId` if the user can provide the earlier build ID from AzDO. +> ❌ **Missing packages on flow PRs β‰  infrastructure.** Flow PRs request *different* packages. Check *which* package and *why* before assuming feed delay. -### Step 3: Verify before claiming +> ❌ **Don't present failures and known issues as separate lists.** Cross-reference them: for each `failedJobDetails` entry, state whether it matches a `knownIssues` entry or is unmatched. An `unclassified` failure can still match a known issue by error pattern. -Before stating a failure's cause, verify your claim: +> ❌ **Don't say "safe to retry" with Build Analysis red.** Map each failing job to a specific known issue first. -- **"Infrastructure failure"** β†’ Did Build Analysis flag it? Does the same test pass on the target branch? If neither, don't call it infrastructure. -- **"Transient/flaky"** β†’ Has it failed before? Is there a known issue? A single non-reproducing failure isn't enough to call it flaky. -- **"PR-related"** β†’ Do the changed files actually relate to the failing test? Correlation in the script output is heuristic, not proof. -- **"Safe to retry"** β†’ Are ALL failures accounted for (known issues or infrastructure), or are you ignoring some? Check the Build Analysis check status β€” if it's red, not all failures are matched. Map each failing job to a specific known issue before concluding "safe to retry." -- **"Not related to this PR"** β†’ Have you checked if the test passes on the target branch? Don't assume β€” verify. +> ❌ **Don't use `Invoke-RestMethod` or `curl` for AzDO/Helix when MCP tools are available.** Check your available tools for `ado-dnceng-public-*`, `ado-dnceng-*`, and `hlx-*` first. REST API fallback is for when MCP tools are genuinely unavailable, not a first resort. ## References -- **Helix artifacts & binlogs**: See [references/helix-artifacts.md](references/helix-artifacts.md) -- **Binlog comparison (passing vs failing)**: See [references/binlog-comparison.md](references/binlog-comparison.md) -- **Build progression (commit-to-build correlation)**: See [references/build-progression-analysis.md](references/build-progression-analysis.md) -- **Subagent delegation patterns**: See [references/delegation-patterns.md](references/delegation-patterns.md) -- **Azure CLI deep investigation**: See [references/azure-cli.md](references/azure-cli.md) -- **Manual investigation steps**: See [references/manual-investigation.md](references/manual-investigation.md) -- **SQL tracking for investigations**: See [references/sql-tracking.md](references/sql-tracking.md) -- **AzDO/Helix details**: See [references/azdo-helix-reference.md](references/azdo-helix-reference.md) +- **Script modes & parameters**: [references/script-modes.md](references/script-modes.md) +- **Failure interpretation**: [references/failure-interpretation.md](references/failure-interpretation.md) +- **Recommendation generation**: [references/recommendation-generation.md](references/recommendation-generation.md) +- **Analysis workflow (Steps 1–3)**: [references/analysis-workflow.md](references/analysis-workflow.md) +- **Helix artifacts & binlogs**: [references/helix-artifacts.md](references/helix-artifacts.md) +- **Binlog comparison**: [references/binlog-comparison.md](references/binlog-comparison.md) +- **Build progression analysis**: [references/build-progression-analysis.md](references/build-progression-analysis.md) +- **Subagent delegation**: [references/delegation-patterns.md](references/delegation-patterns.md) +- **Azure CLI investigation**: [references/azure-cli.md](references/azure-cli.md) +- **Manual investigation**: [references/manual-investigation.md](references/manual-investigation.md) +- **SQL tracking**: [references/sql-tracking.md](references/sql-tracking.md) +- **AzDO/Helix details**: [references/azdo-helix-reference.md](references/azdo-helix-reference.md) ## Tips -1. Check if same test fails on the target branch before assuming transient +1. Check if same test fails on target branch before assuming transient 2. Look for `[ActiveIssue]` attributes for known skipped tests 3. Use `-SearchMihuBot` for semantic search of related issues -4. Use binlog analysis tools to search binlogs for Helix job IDs, build errors, and properties -5. `gh pr checks --json` valid fields: `bucket`, `completedAt`, `description`, `event`, `link`, `name`, `startedAt`, `state`, `workflow` β€” no `conclusion` field, `state` has `SUCCESS`/`FAILURE` directly -6. "Canceled" β‰  "Failed" β€” canceled jobs may have recoverable Helix results. Check artifacts before concluding results are lost. +4. `gh pr checks --json` fields: `bucket`, `completedAt`, `description`, `event`, `link`, `name`, `startedAt`, `state`, `workflow` β€” `state` has `SUCCESS`/`FAILURE` directly (no `conclusion` field) +5. "Canceled" β‰  "Failed" β€” canceled jobs may have recoverable Helix results. Helix data may persist even when AzDO builds have expired β€” query Helix directly if you have job IDs. diff --git a/.github/skills/ci-analysis/references/analysis-workflow.md b/.github/skills/ci-analysis/references/analysis-workflow.md new file mode 100644 index 00000000000000..9a976f616390c9 --- /dev/null +++ b/.github/skills/ci-analysis/references/analysis-workflow.md @@ -0,0 +1,39 @@ +# Analysis Workflow (Steps 1–3) + +After completing Step 0 (Gather Context β€” see SKILL.md), follow these steps. + +## Step 1: Run the Script + +Run with `-ShowLogs` for detailed failure info. See [script-modes.md](script-modes.md) for parameter details. + +## Step 1b: Investigate with AzDO Tools + +When the script output is insufficient (e.g., build timeline fetch fails), use AzDO MCP tools to query builds directly. **Match the org from the build URL to the correct AzDO tools** β€” see [azdo-helix-reference.md](azdo-helix-reference.md#azure-devops-organizations). PR builds are in `dnceng-public`; internal builds are in `dnceng`. + +## Step 2: Analyze Results + +1. **Check Build Analysis** β€” If the Build Analysis GitHub check is **green**, all failures matched known issues and it's safe to retry. If it's **red**, some failures are unaccounted for β€” you must identify which failing jobs are covered by known issues and which are not. For 3+ failures, use SQL tracking to avoid missed matches (see [sql-tracking.md](sql-tracking.md)). +2. **Correlate with PR changes** β€” Same files failing = likely PR-related +3. **Compare with baseline** β€” If a test passes on the target branch but fails on the PR, compare Helix binlogs. See [binlog-comparison.md](binlog-comparison.md) β€” **delegate binlog download/extraction to subagents** to avoid burning context on mechanical work. +4. **Check build progression** β€” If the PR has multiple builds (multiple pushes), check whether earlier builds passed. A failure that appeared after a specific push narrows the investigation to those commits. See [build-progression-analysis.md](build-progression-analysis.md). Present findings as facts, not fix recommendations. +5. **Interpret patterns** (but don't jump to conclusions): + - Same error across many jobs β†’ Real code issue + - Build Analysis flags a known issue β†’ That *specific failure* is safe to retry (but others may not be) + - Failure is **not** in Build Analysis β†’ Investigate further before assuming transient + - Device failures, Docker pulls, network timeouts β†’ *Could* be infrastructure, but verify against the target branch first + - Test timeout but tests passed β†’ Executor issue, not test failure +6. **Check for mismatch with user's question** β€” The script only reports builds for the current head SHA. If the user asks about a job, error, or cancellation that doesn't appear in the results, **ask** if they're referring to a prior build. Common triggers: + - User mentions a canceled job but `canceledJobNames` is empty + - User says "CI is failing" but the latest build is green + - User references a specific job name not in the current results + Offer to re-run with `-BuildId` if the user can provide the earlier build ID from AzDO. + +## Step 3: Verify Before Claiming + +Before stating a failure's cause, verify your claim: + +- **"Infrastructure failure"** β†’ Did Build Analysis flag it? Does the same test pass on the target branch? If neither, don't call it infrastructure. +- **"Transient/flaky"** β†’ Has it failed before? Is there a known issue? A single non-reproducing failure isn't enough to call it flaky. +- **"PR-related"** β†’ Do the changed files actually relate to the failing test? Correlation in the script output is heuristic, not proof. +- **"Safe to retry"** β†’ Are ALL failures accounted for (known issues or infrastructure), or are you ignoring some? Check the Build Analysis check status β€” if it's red, not all failures are matched. Map each failing job to a specific known issue before concluding "safe to retry." +- **"Not related to this PR"** β†’ Have you checked if the test passes on the target branch? Don't assume β€” verify. diff --git a/.github/skills/ci-analysis/references/azdo-helix-reference.md b/.github/skills/ci-analysis/references/azdo-helix-reference.md index ace39f0932eeb6..12aa947e028391 100644 --- a/.github/skills/ci-analysis/references/azdo-helix-reference.md +++ b/.github/skills/ci-analysis/references/azdo-helix-reference.md @@ -31,13 +31,16 @@ Each repository has its own build definition IDs. Here are common ones for dotne ## Azure DevOps Organizations -**Public builds (default):** -- Organization: `dnceng-public` -- Project: `cbb18261-c48f-4abb-8651-8cdcb5474649` +Builds live in two Azure DevOps organizations. **Determine the org from the build URL**, then use AzDO tools for that org: -**Internal/private builds:** -- Organization: `dnceng` -- Project GUID: Varies by pipeline +| Organization | URL pattern | Project | When used | +|---|---|---|---| +| `dnceng-public` | `dev.azure.com/dnceng-public/...` | `public` (GUID: `cbb18261-c48f-4abb-8651-8cdcb5474649`) | PR validation builds (most common) | +| `dnceng` | `dev.azure.com/dnceng/...` | `internal` or varies | Official/internal builds, signed builds | + +**How to pick the right tools:** Look at the build URL from `gh pr checks` output or the script's `[CI_ANALYSIS_SUMMARY]`. The URL contains the org name (e.g., `dev.azure.com/dnceng-public/...`). Search your available AzDO tools for ones matching that org name β€” there may be multiple sets of AzDO tools for different organizations. If a query returns null, you're likely using tools for the wrong org. + +> ⚠️ **Common mistake:** Wrong org = null results. PR validation builds are almost always in `dnceng-public`. If you get null, check the org before trying other approaches. Override with: ```powershell diff --git a/.github/skills/ci-analysis/references/delegation-patterns.md b/.github/skills/ci-analysis/references/delegation-patterns.md index b7dd706fca947b..cd1437d15e60ac 100644 --- a/.github/skills/ci-analysis/references/delegation-patterns.md +++ b/.github/skills/ci-analysis/references/delegation-patterns.md @@ -85,8 +85,10 @@ Check if canceled job "{JOB_NAME}" from build {BUILD_ID} has recoverable Helix r Steps: 1. Check if TRX test results are available for the work item. Parse them for pass/fail counts. -2. If no structured results, check for testResults.xml -3. Parse the XML for pass/fail counts on the element + NOTE: hlx_test_results may error if no TRX files were uploaded (common for non-.NET-test work items like mobile device tests). If it errors, proceed to step 2. +2. If no structured results, search for testResults.xml using hlx_search_file(jobId, workItem, fileName="testResults.xml", pattern="total") +3. If found, parse the XML for pass/fail counts on the element +4. If neither is available, check the console log for test summary lines using hlx_search_log(jobId, workItem, pattern="passed") Return JSON: { "jobName": "...", "hasResults": true, "passed": N, "failed": N } Or: { "jobName": "...", "hasResults": false, "reason": "no testResults.xml uploaded" } diff --git a/.github/skills/ci-analysis/references/failure-interpretation.md b/.github/skills/ci-analysis/references/failure-interpretation.md new file mode 100644 index 00000000000000..afa019d8060779 --- /dev/null +++ b/.github/skills/ci-analysis/references/failure-interpretation.md @@ -0,0 +1,50 @@ +# Interpreting CI Results + +## Result Categories + +**Known Issues section**: Failures matching existing GitHub issues β€” these are tracked and being investigated. + +**Build Analysis check status**: The "Build Analysis" GitHub check is **green** only when *every* failure is matched to a known issue. If it's **red**, at least one failure is unaccounted for β€” do NOT claim "all failures are known issues" just because some known issues were found. You must verify each failing job is covered by a specific known issue before calling it safe to retry. + +**Canceled/timed-out jobs**: Jobs canceled due to earlier stage failures or AzDO timeouts. Dependency-canceled jobs don't need investigation. **Timeout-canceled jobs may have all-passing Helix results** β€” the "failure" is just the AzDO job wrapper timing out, not actual test failures. To verify: use `hlx_status` on each Helix job in the timed-out build (include passed work items). If all work items passed, the build effectively passed. + +> ❌ **Don't dismiss timed-out builds.** A build marked "failed" due to a 3-hour AzDO timeout can have 100% passing Helix work items. Check before concluding it failed. + +**PR Change Correlation**: Files changed by PR appearing in failures β€” likely PR-related. + +**Build errors**: Compilation failures need code fixes. + +**Helix failures**: Test failures on distributed infrastructure. + +**Local test failures**: Some repos (e.g., dotnet/sdk) run tests directly on build agents. These can also match known issues β€” search for the test name with the "Known Build Error" label. + +## Per-Failure Details + +`failedJobDetails` in JSON: Each failed job includes `errorCategory`, `errorSnippet`, and `helixWorkItems`. Use these for per-job classification instead of applying a single `recommendationHint` to all failures. + +Error categories: `test-failure`, `build-error`, `test-timeout`, `crash` (exit codes 139/134/-4), `tests-passed-reporter-failed` (all tests passed but reporter crashed β€” genuinely infrastructure), `unclassified` (investigate manually). + +> ⚠️ **`crash` does NOT always mean tests failed.** Exit code -4 often means the Helix work item wrapper timed out *after* tests completed. Always check `testResults.xml` before concluding a crash is a real failure. See [Recovering Results](#recovering-results-from-crashedcanceled-jobs) below. + +> ⚠️ **Be cautious labeling failures as "infrastructure."** Only conclude infrastructure with strong evidence: Build Analysis match, identical failure on target branch, or confirmed outage. Exception: `tests-passed-reporter-failed` is genuinely infrastructure. + +> ❌ **Missing packages on flow PRs β‰  infrastructure.** Flow PRs can cause builds to request *different* packages. Check *which* package and *why* before assuming feed delay. + +## Recovering Results from Crashed/Canceled Jobs + +When an AzDO job is canceled (timeout) or Helix work items show `Crash` (exit code -4), the tests may have actually passed. Follow this procedure: + +1. **Find the Helix job IDs** β€” Read the AzDO "Send to Helix" step log and search for lines containing `Sent Helix Job`. Extract the job GUIDs. + +2. **Check Helix job status** β€” Get pass/fail summary for each job. Look at `failedCount` vs `passedCount`. + +3. **For work items marked Crash/Failed** β€” Check if tests actually passed despite the crash. Try structured test results first (TRX parsing), then search for pass/fail counts in result files without downloading, then download as last resort: + - Parse the XML: `total`, `passed`, `failed` attributes on the `` element + - If `failed=0` and `passed > 0`, the tests passed β€” the "crash" is the wrapper timing out after test completion + +4. **Verdict**: + - All work items passed or crash-with-passing-results β†’ **Tests effectively passed.** The failure is infrastructure (wrapper timeout). + - Some work items have `failed > 0` in testResults.xml β†’ **Real test failures.** Investigate those specific tests. + - No testResults.xml uploaded β†’ Tests may not have run at all. Check console logs for errors. + +> This pattern is common with long-running test suites (e.g., WasmBuildTests) where tests complete but the Helix work item wrapper exceeds its timeout during result upload or cleanup. diff --git a/.github/skills/ci-analysis/references/manual-investigation.md b/.github/skills/ci-analysis/references/manual-investigation.md index b3b319eaacb9d2..8c8d98c01b12d7 100644 --- a/.github/skills/ci-analysis/references/manual-investigation.md +++ b/.github/skills/ci-analysis/references/manual-investigation.md @@ -36,9 +36,29 @@ $logContent = Invoke-RestMethod -Uri "https://dev.azure.com/dnceng-public/cbb182 $logContent | Select-String -Pattern "error|FAIL" -Context 2,5 ``` +## Search Helix Logs and Artifacts Remotely + +> πŸ’‘ **Prefer remote search over download.** `hlx_search_log` and `hlx_search_file` let you find errors in Helix console logs and uploaded files without downloading them first. Use these before falling back to `hlx_logs` (full log) or `hlx_download` (file download). + +``` +# Search a work item's console log for error patterns +hlx_search_log(jobId, workItem, pattern="error", contextLines=3) + +# Search an uploaded file (e.g., testResults.xml) for specific text +hlx_search_file(jobId, workItem, fileName="testResults.xml", pattern="Failed") + +# Find which work items have specific file types +hlx_find_files(jobId, pattern="*.binlog") +``` + +These tools return matching lines with context β€” much faster than downloading full logs and grepping locally. Use them for: +- Finding specific error messages across large console logs +- Searching test result files for failure patterns +- Locating crash dumps or binlogs across work items + ## Query Helix APIs -> πŸ’‘ **Prefer MCP tools when available** β€” they handle most Helix queries without manual curl commands. Use the APIs below only as fallback. +> πŸ’‘ **Prefer MCP tools when available** β€” `hlx_search_log`, `hlx_search_file`, `hlx_status`, `hlx_logs`, `hlx_files`, etc. handle most Helix queries without manual curl commands. Use the APIs below only as fallback. ```bash # Get job details diff --git a/.github/skills/ci-analysis/references/recommendation-generation.md b/.github/skills/ci-analysis/references/recommendation-generation.md new file mode 100644 index 00000000000000..6fbe2d992cb226 --- /dev/null +++ b/.github/skills/ci-analysis/references/recommendation-generation.md @@ -0,0 +1,60 @@ +# Generating Recommendations + +After the script outputs the `[CI_ANALYSIS_SUMMARY]` JSON block, **you** synthesize recommendations. Do not parrot the JSON β€” reason over it. + +## Decision Logic + +Read `recommendationHint` as a starting point, then layer in context: + +| Hint | Action | +|------|--------| +| `BUILD_SUCCESSFUL` | No failures. Confirm CI is green. | +| `KNOWN_ISSUES_DETECTED` | Known tracked issues found β€” but this does NOT mean all failures are covered. Check the Build Analysis check status: if it's red, some failures are unmatched. Only recommend retry for failures that specifically match a known issue; investigate the rest. | +| `LIKELY_PR_RELATED` | Failures correlate with PR changes. Lead with "fix these before retrying" and list `correlatedFiles`. | +| `POSSIBLY_TRANSIENT` | Failures could not be automatically classified β€” does NOT mean they are transient. Use `failedJobDetails` to investigate each failure individually. | +| `REVIEW_REQUIRED` | Could not auto-determine cause. Review failures manually. | +| `MERGE_CONFLICTS` | PR has merge conflicts β€” CI won't run. Tell the user to resolve conflicts. Offer to analyze a previous build by ID. | +| `NO_BUILDS` | No AzDO builds found (CI not triggered). Offer to check if CI needs to be triggered or analyze a previous build. | + +## Layering Nuance + +Then layer in nuance the heuristic can't capture: + +- **Mixed signals**: Some failures match known issues AND some correlate with PR changes β†’ separate them. Known issues = safe to retry; correlated = fix first. +- **Canceled jobs with recoverable results**: If `canceledJobNames` is non-empty, mention that canceled jobs may have passing Helix results (see [failure-interpretation.md](failure-interpretation.md) β€” Recovering Results). +- **Build still in progress**: If `lastBuildJobSummary.pending > 0`, note that more failures may appear. +- **Multiple builds**: If `builds` has >1 entry, `lastBuildJobSummary` reflects only the last build β€” use `totalFailedJobs` for the aggregate count. +- **BuildId mode**: `knownIssues` and `prCorrelation` won't be populated. Say "Build Analysis and PR correlation not available in BuildId mode." + +## How to Retry + +- **AzDO builds**: Comment `/azp run {pipeline-name}` on the PR (e.g., `/azp run dotnet-sdk-public`) +- **All pipelines**: Comment `/azp run` to retry all failing pipelines +- **Helix work items**: Cannot be individually retried β€” must re-run the entire AzDO build + +## Tone and Output Format + +Be direct. Lead with the most important finding. Structure your response as: +1. **Summary verdict** (1-2 sentences) β€” Is CI green? Failures PR-related? Known issues? +2. **Failure summary table** β€” narrow columns only (Job, Verdict, Issue). Keep job names short (drop redundant prefixes like `runtime-dev-innerloop`). Do NOT put error text or work item lists in the table β€” those go in the detail section below. +3. **Failure details** β€” one bullet per failure with error description, affected work items, and evidence +4. **Recommended actions** (numbered) β€” retry, fix, investigate. Include `/azp run` commands. + +**Use markdown links** for PRs, issues, builds, and jobs so they're clickable: `[#121195](url)`, `[Build 1305302](url)`, `[job name](azdo-job-url)`. The script output and MCP tools provide the URLs β€” thread them through to your response. + +Example layout for step 2+3: + +``` +| # | Job | Verdict | Issue | +|---|-----|---------|-------| +| 1 | [browser-wasm linux Release WasmBuildTests](https://dev.azure.com/…) | Known flaky βœ… | [#121195](https://github.com/dotnet/runtime/issues/121195) | +| 2 | [linux-x64 Debug Mono Interpreter LibTests](https://dev.azure.com/…) | Known flaky βœ… | [#100800](https://github.com/dotnet/runtime/issues/100800) | +| 3 | [coreclr Pri0 Runtime Tests linux x64](https://dev.azure.com/…) | Known flaky βœ… | [#110173](https://github.com/dotnet/runtime/issues/110173) | + +**Details:** +- **#1**: Playwright.TargetClosedException + dbus socket missing in WBT-NoWorkload +- **#2**: 8 work items (ComInterfaceGen, IntrinsicsInSPC, JSImportGen, …) β€” all exit code 139 (SIGSEGV), mono interpreter crashes +- **#3**: stackoverflowtester timeout in baseservices-exceptions β€” ASSERT: "Target stack has been corrupted" +``` + +Synthesize from: JSON summary (structured facts) + human-readable output (details/logs) + Step 0 context (PR type, author intent). diff --git a/.github/skills/ci-analysis/references/script-modes.md b/.github/skills/ci-analysis/references/script-modes.md new file mode 100644 index 00000000000000..31164e14dda902 --- /dev/null +++ b/.github/skills/ci-analysis/references/script-modes.md @@ -0,0 +1,48 @@ +# Script Modes and Parameters + +## Key Parameters + +| Parameter | Description | +|-----------|-------------| +| `-PRNumber` | GitHub PR number to analyze | +| `-BuildId` | Azure DevOps build ID | +| `-ShowLogs` | Fetch and display Helix console logs | +| `-Repository` | Target repo (default: dotnet/runtime) | +| `-MaxJobs` | Max failed jobs to show (default: 5) | +| `-SearchMihuBot` | Search MihuBot for related issues | + +## Three Modes + +The script operates in three distinct modes depending on what information you have: + +| You have... | Use | What you get | +|-------------|-----|-------------| +| A GitHub PR number | `-PRNumber 12345` | Full analysis: all builds, failures, known issues, structured JSON summary | +| An AzDO build ID | `-BuildId 1276327` | Single build analysis: timeline, failures, Helix results | +| A Helix job ID (optionally a specific work item) | `-HelixJob "..." [-WorkItem "..."]` | Deep dive: list work items for the job, or with `-WorkItem`, focus on a single work item's console logs, artifacts, and test results | + +> ❌ **Don't guess the mode.** If the user gives a PR URL, use `-PRNumber`. If they paste an AzDO build link, extract the build ID. If they reference a specific Helix job, use `-HelixJob`. + +## What the Script Does + +### PR Analysis Mode (`-PRNumber`) +1. Discovers AzDO builds associated with the PR (from GitHub check status; for full build history, query AzDO builds on `refs/pull/{PR}/merge` branch) +2. Fetches Build Analysis for known issues +3. Gets failed jobs from Azure DevOps timeline +4. **Separates canceled jobs from failed jobs** (canceled may be dependency-canceled or timeout-canceled) +5. Extracts Helix work item failures from each failed job +6. Fetches console logs (with `-ShowLogs`) +7. Searches for known issues with "Known Build Error" label +8. Correlates failures with PR file changes +9. **Emits structured summary** β€” `[CI_ANALYSIS_SUMMARY]` JSON block with all key facts for the agent to reason over + +> **After the script runs**, you (the agent) generate recommendations. The script collects data; you synthesize the advice. See [recommendation-generation.md](recommendation-generation.md). + +### Build ID Mode (`-BuildId`) +1. Fetches the build timeline directly (skips PR discovery) +2. Performs steps 3–7 from PR Analysis Mode, but does **not** fetch Build Analysis known issues or correlate failures with PR file changes (those require a PR number). Still emits `[CI_ANALYSIS_SUMMARY]` JSON. + +### Helix Job Mode (`-HelixJob` [and optional `-WorkItem`]) +1. With `-HelixJob` alone: enumerates work items for the job and summarizes their status +2. With `-HelixJob` and `-WorkItem`: queries the specific work item for status and artifacts +3. Fetches console logs and file listings, displays detailed failure information diff --git a/.github/skills/pr-triage/SKILL.md b/.github/skills/pr-triage/SKILL.md new file mode 100644 index 00000000000000..1f85fae0ef53e1 --- /dev/null +++ b/.github/skills/pr-triage/SKILL.md @@ -0,0 +1,329 @@ +--- +name: pr-triage +description: > + Triage open PRs for merge readiness in dotnet/runtime. Determine next action for each PR + and rank PRs by how close they are to merging. + USE FOR: "which PRs are ready to merge", "what's blocking PR #X", "triage open PRs", + "next action for PR", "show merge-ready PRs", "triage area-X PRs", "show approved PRs", + "triage community contributions", "PR merge readiness". + DO NOT USE FOR: CI failure deep-dive (use ci-analysis skill), code review for correctness + (use code-review skill), codeflow PR health (use vmr-codeflow-status skill), + performance benchmarking (use performance-benchmark skill). +--- + +# PR Triage Skill + +Identify which pull requests in dotnet/runtime are ready to merge and determine the next action for each PR. This is a **read-only analysis skill** β€” it never modifies PRs, issues, labels, or comments. + +**Community contributions** are flagged so maintainers can make informed prioritization +decisions. Timely feedback β€” even a quick "not right now" β€” respects contributors' time. + +## Architecture + +All data fetching and scoring is done by a PowerShell script. Your job as the AI is: + +1. **Parse** the user's request to determine filters (label, author, limit, etc.) +2. **Run** the script with the right flags +3. **Format** the JSON output as a readable table +4. **Annotate** with any additional context the user asked for + +The script handles: batched GraphQL, Build Analysis extraction, review/thread parsing, 12-dimension scoring, next-action determination, and "Who" identification β€” all in ~15 seconds for 50 PRs. + +## When to Use This Skill + +- "Which PRs are ready to merge?" / "Triage area-X PRs" / "What's blocking PR #X?" +- "What PRs need my attention?" / "Show approved PRs" / "Triage community contributions" + +## Batch Mode (Primary) + +### Step 1 β€” Map User Request to Script Flags + +| User says | Script flag | +|-----------|-------------| +| "triage area-CodeGen-coreclr" | `-Label "area-CodeGen-coreclr"` | +| "triage all open PRs" | (no -Label) `-Limit 300` | +| "top 20 PRs" | `-Top 20` | + +### Step 2 β€” Estimate and Run the Script + +> ⚠️ **IMPORTANT**: You MUST tell the user the estimated time BEFORE running the script. +> Print the estimate as your first output, then run the script. Never skip this step. + +The script is dominated by GraphQL batching at ~2.5 seconds per 10 PRs, plus ~5s overhead. +Note that `-MyActions` and `-NextAction` are post-filters β€” the script still scans all +PRs matching the pre-fetch filters, so use the pre-fetch count for the estimate. + +| Query type | Typical PRs scanned | Estimated time | +|-----------|-------------|---------------| +| Single PR (`-PrNumber`) | 1 | ~5 seconds | +| `-Label area-X` | 10-60 | 10-20 seconds | +| `-Label area-X` + `-MyActions` | 10-60 (same scan) | 10-20 seconds | +| Full repo (`-Limit 500`) | ~300 | 60-90 seconds | +| Full repo + any post-filter | ~300 (same scan) | 60-90 seconds | + +Example: *"This will scan ~300 PRs and filter to your action items β€” should take about 60-90 seconds."* + +```powershell +.\.github\skills\pr-triage\scripts\Get-PrTriageData.ps1 -Label "area-CodeGen-coreclr" +``` + +The script outputs JSON with this structure: + +```json +{ + "timestamp": "...", + "repo": "dotnet/runtime", + "filters": { "label": "area-CodeGen-coreclr", "author": null, "top": 0, "..." : "..." }, + "scanned": 58, + "analyzed": 46, + "returned": 46, + "screened_out": { "drafts_count": 12, "drafts": ["...first 10..."], "bots": [...], "needs_author_action": [...], "stale": [...] }, + "quick_actions": { "ready_to_merge": 1, "needs_maintainer_review": 10, "needs_author_action": 35, "blocked_conflicts": 9 }, + "prs": [ + { + "number": 123546, "title": "...", "author": "jonathandavies-arm", + "score": 8.2, "ci": "SUCCESS", "ci_detail": "91/2/0", + "unresolved_threads": 0, "total_threads": 1, "total_comments": 3, "distinct_commenters": 2, + "mergeable": "MERGEABLE", + "approval_count": 1, "is_community": true, + "age_days": 35, "days_since_update": 2, "changed_files": 3, "lines_changed": 45, + "next_action": "Ready to merge", + "who": "@EgorBo", + "blockers": "β€”", + "why": "CI:pass, OwnerApproved, Small" + } + ] +} +``` + +### Step 3 β€” Format and Annotate + +Format the JSON as a clean markdown table with a brief summary and observations. +This is a cheap LLM pass β€” the JSON is small and the output is a formatted table plus +a few sentences. Use `ci_detail` as `pass/fail/run` and map `ci` to emoji. + +**Column order:** Score | PR | Title | Who | Next Action | CI | Disc | Age | Updated | Files | Author (last) + +**PR column:** Always hyperlink: `[#12345](https://github.com/{repo}/pull/12345)` + +**CI emoji:** SUCCESSβ†’βœ…, FAILUREβ†’βŒ, IN_PROGRESS→⏳, ABSENTβ†’βš οΈ + +**Discussion column:** Show as `{total_threads}t/{distinct_commenters}p` (threads/people). +Heavy discussion (>15t or >5p) is a signal the PR is expensive to push forward. + +Example output: + +```markdown +## Merge Readiness: area-CodeGen-coreclr +Scanned 58 β†’ 46 analyzed (12 drafts excluded) + +| Score | PR | Title | Who | Next Action | CI | Disc | Age | Updated | Files | Author | +|------:|---:|-------|-----|-------------|----|-----:|----:|--------:|------:|--------| +| 8.4 | [#123546](https://github.com/dotnet/runtime/pull/123546) | arm64: Remove widening casts before truncating | @EgorBo | Ready to merge βœ… | βœ… 91/2/0 | 1t/2p | 5d | 1d | 3 | @jonathandavies-arm | +| 7.9 | [#122485](https://github.com/dotnet/runtime/pull/122485) | [RISC-V] Enable instruction printing | @SkyShield | Author: merge main (stale 17d) | βœ… 91/2/0 | 1t/2p | 24d | 17d | 2 | @SkyShield | +| ... | | | | | | | | | | | + +### Summary +- **Ready to merge**: 1 β€” #123546 (approved by @EgorBo, CI green, no threads) +- **Needs maintainer review**: 10 PRs β€” mostly awaiting @JulieLeeMSFT / @BruceForstall +- **Needs author action**: 35 PRs +- **Blocked by conflicts**: 9 PRs + +### Observations +- #124846 has 38 review threads from 5 people β€” heavy discussion, will need significant effort to land +- 9 PRs have merge conflicts; most are older community contributions (>100 days) +- @amanasifkhalid has 6 open PRs, several with conflicts β€” may benefit from a triage pass +- #124953 by @akoeplinger is a simple test cleanup (0 threads, CI green) β€” quick win for a reviewer +``` + +**Key guidelines for observations:** +- Keep to 3-5 bullet points β€” highlight what's actionable +- Call out quick wins (high score, no threads, just needs a reviewer click) +- Call out expensive PRs (heavy discussion, many conflicts, large diffs) +- Note patterns (one author with many stale PRs, cluster of PRs needing same reviewer) +- **For community PRs**: note that they may need more shepherding and may not align with current investment. Frame feedback constructively β€” even a quick "not right now" respects the contributor's time +- Don't repeat what's already visible in the table + +### Step 4 β€” Cache Results for Follow-up Queries + +After a batch run, load the results into a SQL table so follow-up questions don't require +re-running the script. Create the table if it doesn't exist, then load from JSON: + +```sql +CREATE TABLE IF NOT EXISTS pr_analysis ( + number INTEGER PRIMARY KEY, title TEXT, author TEXT, score REAL, + ci TEXT, ci_detail TEXT, unresolved_threads INTEGER, total_threads INTEGER, + distinct_commenters INTEGER, mergeable TEXT, approval_count INTEGER, + is_community INTEGER, age_days INTEGER, days_since_update INTEGER, + changed_files INTEGER, lines_changed INTEGER, next_action TEXT, + who TEXT, blockers TEXT, why TEXT +); +``` + +Then parse the JSON `prs` array and INSERT each row. Alternatively, run the script with +`-OutputCsv` to get tab-separated output and load that. + +Once loaded, answer follow-up questions with SQL queries β€” no API calls needed: +- "Which community PRs need review?" β†’ `WHERE is_community = 1 AND next_action LIKE '%review%'` +- "Who has the most open PRs?" β†’ `GROUP BY author ORDER BY COUNT(*) DESC` +- "Show PRs with heavy discussion" β†’ `WHERE total_threads > 15 ORDER BY total_threads DESC` +- "What needs @danmoseley's attention?" β†’ `WHERE who LIKE '%danmoseley%'` + +### Step 5 β€” Sentiment Check for Top PRs (Optional) + +For the top 3-5 PRs in the results (highest score), fetch recent comments to assess +reviewer sentiment. This adds color the score can't capture β€” e.g., "LGTM ship it" vs +"approved but I have concerns about the approach". + +```bash +gh pr view {number} --repo dotnet/runtime --json comments --jq '.comments[-5:][] | .author.login + ": " + .body[:200]' +``` + +Use this to add a brief **sentiment note** in the Observations section, such as: +- "#123546 β€” @EgorBo said 'looks good, ready to merge' β€” strong positive signal" +- "#124663 β€” ongoing design discussion between @tannergooding and author, approval may be conditional" + +**Guidelines:** +- Only do this for the top 3-5 PRs β€” not the full list +- Focus on the most recent 3-5 comments per PR +- Look for: explicit merge-readiness signals, conditional approvals, soft blocks ("let's wait for design review"), or enthusiastic endorsement +- Don't attempt to score or quantify sentiment β€” just note it in observations +- Skip this step if the user asked for a quick summary or broad overview + +### Step 6 β€” Answer Follow-ups + +If the user asks about a specific PR from the results, query the SQL table first. +If they ask for CI failure details, delegate to the **ci-analysis** skill. + +## Single-PR Mode + +For "what's blocking PR #12345?", run the script and filter to that PR number, +then present the detailed breakdown using all available fields (score, ci, ci_detail, +unresolved_threads, mergeable, approval_count, blockers, why, who). + +For deeper analysis (context drift, breaking change risk, perf-sensitive areas), +supplement with: + +```bash +gh pr view {number} --repo dotnet/runtime --json files +``` + +And check the [merge-readiness rubric](references/merge-readiness-rubric.md) for +dimensions 8, 14-16 which require file-level analysis. + +## Script Parameters + +Map the user's request to these flags. Combine as needed. + +### Pre-fetch filters (reduce API calls) + +| Parameter | Default | Description | Example user request | +|-----------|---------|-------------|---------------------| +| `-Label` | (none) | Area label filter | "triage area-System.Net PRs" | +| `-Author` | (none) | Filter to specific PR author | "show PRs by @stephentoub" | +| `-Assignee` | (none) | Filter to specific assignee | "PRs assigned to me" | +| `-Limit` | 500 | Max PRs from `gh pr list` | "scan all PRs" β†’ `-Limit 500` | +| `-Repo` | `dotnet/runtime` | Repository | (rarely changed) | + +### Inclusion/exclusion toggles + +| Parameter | Default | Description | Example user request | +|-----------|---------|-------------|---------------------| +| `-Community` | off | Only community-contribution PRs | "triage community contributions" | +| `-IncludeDrafts` | off | Include draft PRs | "show all PRs including drafts" | +| `-ExcludeCopilot` | off | Exclude copilot-swe-agent PRs | "skip bot PRs" | +| `-IncludeNeedsAuthor` | off | Include needs-author-action PRs | "show everything including needs-author" | +| `-IncludeStale` | off | Include no-recent-activity PRs | "show stale PRs too" | +| `-HasLabel` | (none) | Require a specific label | "show api-approved PRs" | +| `-ExcludeLabel` | (none) | Exclude a specific label | "skip blocked PRs" | + +### Age and recency filters + +| Parameter | Default | Description | Example user request | +|-----------|---------|-------------|---------------------| +| `-MinAge` | 0 | Minimum PR age in days | "PRs older than 30 days" β†’ `-MinAge 30` | +| `-MaxAge` | 0 | Maximum PR age in days | "PRs less than a week old" β†’ `-MaxAge 7` | +| `-UpdatedWithin` | 0 | Updated within N days | "recently active PRs" β†’ `-UpdatedWithin 7` | + +### Post-scoring filters (applied after scoring) + +| Parameter | Default | Description | Example user request | +|-----------|---------|-------------|---------------------| +| `-MinApprovals` | 0 | Minimum approval count | "show approved PRs" β†’ `-MinApprovals 1` | +| `-MinScore` | 0 | Minimum score (0-10) | "show PRs scoring 7+" β†’ `-MinScore 7` | +| `-NextAction` | (none) | Filter by action type: `ready`, `review`, `author`, `conflicts`, `ci` | "which are ready to merge?" β†’ `-NextAction ready` | +| `-MyActions` | (none) | Show PRs where this person owns next step | "what needs my attention?" β†’ `-MyActions @danmoseley` | +| `-PrNumber` | (none) | Single PR mode | "what's blocking #12345?" β†’ `-PrNumber 12345` | +| `-Top` | 0 (all) | Return only top N results | "show top 10" β†’ `-Top 10` | +| `-OutputCsv` | off | Tab-separated output for SQL import | (used internally for caching) | + +### Example invocations + +```powershell +# Triage an area +.\Get-PrTriageData.ps1 -Label "area-CodeGen-coreclr" + +# What's ready to merge across the whole repo? +.\Get-PrTriageData.ps1 -Limit 300 -NextAction ready + +# Community PRs needing maintainer review +.\Get-PrTriageData.ps1 -Community -NextAction review + +# My action items +.\Get-PrTriageData.ps1 -Limit 300 -MyActions "@danmoseley" + +# High-scoring PRs updated recently +.\Get-PrTriageData.ps1 -Limit 300 -MinScore 7 -UpdatedWithin 7 + +# Single PR deep-dive +.\Get-PrTriageData.ps1 -PrNumber 123546 + +# Old community PRs that are almost ready +.\Get-PrTriageData.ps1 -Community -MinAge 30 -MinScore 5 + +# Top 15 PRs with at least one approval +.\Get-PrTriageData.ps1 -Limit 300 -MinApprovals 1 -Top 15 +``` + +## Score Dimensions (0-10 scale) + +The script scores 12 dimensions with weighted composite (see [rubric](references/merge-readiness-rubric.md)): + +| Weight | Dimension | What it measures | +|--------|-----------|-----------------| +| 3.0 | CI (Build Analysis) | Hard blocker β€” can't merge if BA is red | +| 3.0 | Conflicts | Hard blocker β€” unmergeable | +| 3.0 | Maintainer Review | Hard blocker β€” runtime requires owner/triager approval | +| 2.0 | Feedback | Unresolved review threads | +| 2.0 | Approval Strength | Who approved: area owner > triager > community | +| 1.5 | Staleness | Days since last update | +| 1.5 | Discussion Complexity | Thread count and distinct commenters | +| 1.0 | Alignment | Has area label, not untriaged | +| 1.0 | Freshness | Recent activity | +| 1.0 | Size | Smaller = easier to review | +| 0.5 | Community | Flags community PRs for visibility β€” they need different handling | +| 0.5 | Velocity | Review momentum | + +## "My Actions" Filter + +When the user asks "what PRs need my attention?", run the script then filter results +where `who` contains their username. The script already identifies the responsible +person(s) for each PR using area ownership, review threads, and approval state. + +## Anti-Patterns + +> 🚨 **Never approve or request changes on PRs.** Read-only analysis only. + +> ❌ **Don't deep-dive CI failures** β€” delegate to the **ci-analysis** skill. + +> ❌ **Don't review code for correctness** β€” delegate to the **code-review** skill. + +> ❌ **Don't modify PRs, issues, labels, or comments.** + +> ⚠️ **Do NOT recommend rebase/merge-main just because a build is a few days old.** CI takes 2–3 hours. Only recommend it when build is >14 days old. + +## References + +- **Scoring details**: [references/merge-readiness-rubric.md](references/merge-readiness-rubric.md) +- **Batch workflow**: [references/batch-analysis-workflow.md](references/batch-analysis-workflow.md) +- **Runtime signals**: [references/runtime-signals.md](references/runtime-signals.md) diff --git a/.github/skills/pr-triage/references/batch-analysis-workflow.md b/.github/skills/pr-triage/references/batch-analysis-workflow.md new file mode 100644 index 00000000000000..d1de0b412d90bf --- /dev/null +++ b/.github/skills/pr-triage/references/batch-analysis-workflow.md @@ -0,0 +1,73 @@ +# Batch PR Analysis Workflow + +## Overview + +dotnet/runtime typically has several hundred open PRs. The PowerShell script +`Get-PrTriageData.ps1` handles all batch analysis in a single run: + +1. Fetches PR list via `gh pr list` (1 REST call) +2. Quick-screens out drafts, bots, stale, needs-author-action +3. Fetches reviews, threads, and Build Analysis via batched GraphQL (10 PRs per call) +4. Scores all 12 dimensions and determines next action + who +5. Outputs sorted JSON + +**Performance**: ~2.5 seconds per 10 PRs + ~5s overhead. Full repo (300+ PRs) = 60-90 seconds. + +## Default Exclusions + +The script excludes by default (toggles available to include): + +| Excluded | Why | Include flag | +|----------|-----|-------------| +| Drafts | Not ready for review | `-IncludeDrafts` | +| `dotnet-maestro` / `github-actions` bots | Automated dependency flow | (always excluded) | +| `needs-author-action` label | Ball is in author's court | `-IncludeNeedsAuthor` | +| `no-recent-activity` label | At risk of auto-close | `-IncludeStale` | + +`copilot-swe-agent` PRs are **included by default** β€” they are maintainer-initiated. +Use `-ExcludeCopilot` to exclude them. + +## GraphQL Batching + +A single GraphQL query fetches reviews, review threads, and Build Analysis +(via `statusCheckRollup`) for up to 10 PRs at once: + +```graphql +{ + repository(owner: "dotnet", name: "runtime") { + pr0: pullRequest(number: 12345) { + number + comments { totalCount } + reviews(last: 10) { nodes { author { login } state } } + reviewThreads(first: 50) { nodes { isResolved comments(first: 5) { nodes { author { login } } } } } + commits(last: 1) { nodes { commit { statusCheckRollup { contexts(first: 100) { nodes { ... on CheckRun { name conclusion status } } } } } } } + } + pr1: pullRequest(number: 12346) { ... } + } +} +``` + +**Constraint**: `contexts(first: 100)` is the GraphQL max. PRs with >100 checks +may not include Build Analysis in the first page (rare). + +## Caching in SQL + +After a batch run, the LLM should load results into a SQL table so follow-up +questions don't require re-running the script. See SKILL.md Step 4 for the schema. + +## API Budget + +| Candidates | GraphQL calls | Total time | +|-----------|--------------|------------| +| 10 PRs | 1 | ~8s | +| 50 PRs | 5 | ~18s | +| 100 PRs | 10 | ~30s | +| 300 PRs | 30 | ~80s | + +GitHub GraphQL limit: 5,000 points/hour. Even 300 PRs is well within limits. + +## Anti-Patterns + +> ❌ Don't try to analyze PRs one-by-one with separate `gh` calls β€” use the script +> ❌ Don't hold all PR data in LLM context β€” use SQL table for follow-ups +> ❌ Don't re-run the script for follow-up queries β€” query the SQL cache instead diff --git a/.github/skills/pr-triage/references/merge-readiness-rubric.md b/.github/skills/pr-triage/references/merge-readiness-rubric.md new file mode 100644 index 00000000000000..7266a557712674 --- /dev/null +++ b/.github/skills/pr-triage/references/merge-readiness-rubric.md @@ -0,0 +1,463 @@ +# Merge-Readiness Rubric + +Reference for evaluating PRs across 16 merge-readiness dimensions and computing +composite scores for ranking. + +Each dimension yields a score: + +| Symbol | Meaning | Points | +|--------|---------|--------| +| βœ… | Ready | 1 | +| ⚠️ | Needs attention | 0.5 | +| ❌ | Blocking | 0 | + +--- + +## Per-Dimension Evaluation + +### 1. CI Status + +**How to check** + +```bash +gh pr checks {PR} --repo dotnet/runtime --json name,state +``` + +Find the **Build Analysis** entry. + +**Scoring** + +| Score | Condition | +|-------|-----------| +| βœ… 1 | Build Analysis green (all failures matched known issues) | +| ⚠️ 0.5 | Build Analysis pending or no builds yet | +| ❌ 0 | Build Analysis red (unmatched failures) | + +**Edge cases** + +- No "Build Analysis" check at all β†’ build may not have run β†’ ⚠️ + +--- + +### 2. Build Staleness + +**How to check** + +Compare `headRefOid` from `gh pr view` against build info from `gh pr checks` +timestamps. + +**Scoring** + +| Score | Condition | +|-------|-----------| +| βœ… 1 | Build ran on current HEAD and within last 3 days | +| ⚠️ 0.5 | Build is 3–14 days old | +| ❌ 0 | Build is >14 days old or ran on a different SHA than current HEAD | + +**Next-action guidance**: Do NOT recommend "merge main / rebase" solely because +a build is a few days old. Running CI takes 2–3 hours, so a rebase is costly. +Only recommend rebase when: +- Build is >14 days old, OR +- Context drift (dimension 8) shows significant changes to touched directories, OR +- Build ran on a different SHA than current HEAD + +A 7-day-old green build with no context drift is fine β€” the next action should +be whatever the other blocking dimensions indicate. + +**Edge cases** + +- PR just pushed, build queued β†’ ⚠️ + +--- + +### 3. Maintainer Review + +**How to check** + +```bash +gh pr view {PR} --repo dotnet/runtime --json reviews +``` + +Match reviewer logins against the **Approval Authority Levels** defined in [runtime-signals.md](runtime-signals.md#approval-authority-levels): + +1. **Area owner/lead** (level 1) β€” parse `docs/area-owners.md`, match reviewer login against Owners column for the PR's area label +2. **Community triager** (level 2) β€” check if reviewer is in the Community Triagers list in `docs/area-owners.md` +3. **Frequent contributor** (level 3) β€” `gh api "repos/dotnet/runtime/commits?author={login}&path={dir}&per_page=5"` β€” 3+ hits + +**Scoring** + +| Score | Condition | +|-------|-----------| +| βœ… 1 | Approved by at least one area owner/lead (level 1) | +| ⚠️ 0.75 | Approved by community triager (level 2) β€” strong signal but not merge authority | +| ⚠️ 0.5 | Approved by frequent contributor (level 3) or commented-only by any reviewer | +| ❌ 0 | No reviews, or only `CHANGES_REQUESTED` from owner | + +**Edge cases** + +- PR has multiple area labels β†’ need approval from at least one area's owner. +- Community triager approval (level 2) is a strong positive signal even though it doesn't fully satisfy the maintainer review requirement β€” note this in the output. + +--- + +### 4. Feedback Addressed + +**How to check** + +```bash +gh pr view {PR} --repo dotnet/runtime --json labels +``` + +Also query GraphQL `reviewThreads` for unresolved threads: + +```graphql +{ + repository(owner: "dotnet", name: "runtime") { + pullRequest(number: PR) { + reviewThreads(first: 100) { + nodes { isResolved } + } + } + } +} +``` + +**Scoring** + +| Score | Condition | +|-------|-----------| +| βœ… 1 | No `needs-author-action` label AND zero unresolved review threads | +| ⚠️ 0.5 | Has unresolved threads but no `needs-author-action` label (label may not have been applied yet) | +| ❌ 0 | `needs-author-action` label present | + +--- + +### 5. Merge Conflicts + +**How to check** + +```bash +gh pr view {PR} --repo dotnet/runtime --json mergeable +``` + +**Scoring** + +| Score | Condition | +|-------|-----------| +| βœ… 1 | `MERGEABLE` | +| ⚠️ 0.5 | `UNKNOWN` (GitHub hasn't computed yet) | +| ❌ 0 | `CONFLICTING` | + +--- + +### 6. Alignment + +**How to check** + +```bash +gh pr view {PR} --repo dotnet/runtime --json labels +``` + +Check for `area-*` label and absence of `untriaged`. + +**Scoring (batch mode)** + +| Score | Condition | +|-------|-----------| +| βœ… 1 | Has area label, not untriaged | +| ❌ 0 | `untriaged` label or no area label | + +**Single-PR mode**: Can additionally check linked issues via GraphQL +`closingIssuesReferences` for a richer signal. + +--- + +### 7. Freshness + +**How to check** + +Check labels for `no-recent-activity` and the `updatedAt` field. + +```bash +gh pr view {PR} --repo dotnet/runtime --json labels,updatedAt +``` + +**Scoring** + +| Score | Condition | +|-------|-----------| +| βœ… 1 | Updated within 14 days, no stale label | +| ⚠️ 0.5 | Updated 14–30 days ago | +| ❌ 0 | `no-recent-activity` label or >30 days since update | + +--- + +### 8. Context Drift (medium cost) + +**How to check** + +Get PR's changed file paths, then for each directory touched, check recent +commits on main since the PR was last updated: + +```bash +gh pr view {PR} --repo dotnet/runtime --json files +gh api "repos/dotnet/runtime/commits?path={dir}&since={PR_updatedAt}&per_page=10" +``` + +**Scoring** + +| Score | Condition | +|-------|-----------| +| βœ… 1 | <5 commits to touched directories on main since PR last updated | +| ⚠️ 0.5 | 5–20 commits (moderate churn) | +| ❌ 0 | >20 commits (significant context shift, even if it merges clean) | + +--- + +### 9. PR Size / Complexity + +**How to check** + +```bash +gh pr view {PR} --repo dotnet/runtime --json changedFiles,additions,deletions,files +``` + +Also derive directory spread: count unique parent directories from +`files[].path`. + +**Scoring** + +| Score | Condition | +|-------|-----------| +| βœ… 1 | ≀5 files, ≀200 lines changed, ≀2 directories | +| ⚠️ 0.5 | 6–20 files, or 200–500 lines, or 3–5 directories | +| ❌ 0 | >20 files, or >500 lines, or >5 directories | + +--- + +### 10. Community Contribution Flag + +**How to check** + +Check for `community-contribution` label; match author login against known team +members. + +```bash +gh pr view {PR} --repo dotnet/runtime --json labels,author +``` + +**Scoring** + +| Score | Condition | +|-------|-----------| +| βœ… 1 | Team member author (no special handling needed) | +| ⚠️ 0.5 | Community contributor β€” flagged for visibility so maintainers can prioritize | + +**Score note**: The lower score reflects that community PRs are typically more +expensive to drive (alignment, mentoring, iteration) β€” not a quality judgment. +The `is_community` flag in output lets maintainers filter and prioritize as they see fit. + +--- + +### 11. Linked Issue Priority + +**How to check** + +Query GraphQL `closingIssuesReferences` to find linked issues, then inspect +each: + +```bash +gh issue view {ISSUE} --repo dotnet/runtime --json labels,milestone +``` + +Look for labels containing `Priority:` or a milestone. + +**Scoring** + +| Score | Condition | +|-------|-----------| +| βœ… 1 | Linked to `Priority:1` issue or current milestone | +| ⚠️ 0.5 | Linked to lower-priority issue | +| ❌ 0 | No linked issue (neutral, doesn't block) | + +--- + +### 12. Approval Strength + +**How to check** + +```bash +gh pr view {PR} --repo dotnet/runtime --json reviews +``` + +Count `APPROVED` reviews and classify each reviewer by approval authority level (see [runtime-signals.md](runtime-signals.md#approval-authority-levels)): +- **Level 1** (area owner/lead): has merge authority +- **Level 2** (community triager): strong signal, weight 1.5Γ— +- **Level 3** (frequent contributor): domain expertise, weight 1Γ— +- **Level 4** (new contributor): valued feedback, weight 0.5Γ— + +**Scoring** + +| Score | Condition | +|-------|-----------| +| βœ… 1 | 2+ approvals with at least one from area owner (tier 1) | +| ⚠️ 0.75 | 1 area owner approval, or approval from community triager (tier 2) + another reviewer | +| ⚠️ 0.5 | 1 approval from community triager alone, or multiple tier 3/4 approvals | +| ❌ 0 | No approvals | + +--- + +### 13. Review Velocity + +**How to check** + +Compare `createdAt`, `updatedAt`, and the latest review timestamp. + +```bash +gh pr view {PR} --repo dotnet/runtime --json createdAt,updatedAt,reviews +``` + +**Scoring** + +| Score | Condition | +|-------|-----------| +| βœ… 1 | PR <7 days old with review activity, or PR with review in last 3 days | +| ⚠️ 0.5 | PR 7–14 days old with some review activity | +| ❌ 0 | PR >14 days old with no reviews (stalling β€” may need proactive outreach) | + +--- + +### 14. Author Familiarity (medium cost) + +**How to check** + +For each directory the PR touches, query recent commits by the same author: + +```bash +gh api "repos/dotnet/runtime/commits?author={author}&path={dir}&per_page=5" +``` + +**Scoring** + +| Score | Condition | +|-------|-----------| +| βœ… 1 | Author has 3+ recent commits in the same area | +| ⚠️ 0.5 | Author has 1–2 commits in the area | +| ❌ 0 | No prior commits from this author in the touched area (first-time contributor to this area) | + +--- + +### 15. Perf-Sensitive Area (expensive, single-PR only) + +**How to check** + +Match changed file paths against known hot paths: + +| Path pattern | Reason | +|-------------|--------| +| `src/libraries/System.Private.CoreLib/` | Perf-critical core library | +| `src/coreclr/jit/` | JIT compiler | +| `src/coreclr/gc/` | Garbage collector | +| `src/libraries/System.Runtime/` | Core runtime | +| Any path containing `Span`, `Memory`, or `Buffer` | Hot-path types | + +Also check if `performance-benchmark` was already run (look for @EgorBot +comments in the PR timeline). + +**Scoring** + +| Score | Condition | +|-------|-----------| +| βœ… 1 | Not perf-sensitive, or benchmark already run | +| ⚠️ 0.5 | Perf-sensitive area, benchmark not yet run | + +This dimension is never blocking (❌), only advisory. + +--- + +### 16. Breaking Change Risk (expensive, single-PR only) + +**How to check** + +Look for `ref/` files in the changed files list (API surface changes). Check +for `api-ready-for-review` or `api-approved` labels. + +```bash +gh pr view {PR} --repo dotnet/runtime --json files,labels +``` + +**Scoring** + +| Score | Condition | +|-------|-----------| +| βœ… 1 | No `ref/` changes, or `api-approved` label present | +| ⚠️ 0.5 | `ref/` changes with `api-ready-for-review` (review pending) | +| ❌ 0 | `ref/` changes without any API review label | + +--- + +## Composite Scoring + +For batch-mode ranking, compute a weighted score per PR. Multiply each +dimension's point value (0, 0.5, or 1) by its weight and sum. + +### Weight Table (as implemented in the script) + +The script computes 12 dimensions. Dimensions 8 (Context Drift), 11 (Linked Issue +Priority), 14 (Author Familiarity), 15 (Perf-Sensitive Area), and 16 (Breaking +Change Risk) are documented above for single-PR deep dives but are **not included** +in the batch composite score. + +| # | Dimension | Weight | Rationale | +|---|-----------|--------|-----------| +| 1 | CI Status | 3 | Can't merge without green CI | +| 5 | Merge Conflicts | 3 | Can't merge with conflicts | +| 3 | Maintainer Review | 3 | Required for merge | +| 4 | Feedback Addressed | 2 | Blocks merge if outstanding | +| 12 | Approval Strength | 2 | Stronger approvals = closer to merge | +| 2 | Staleness | 1.5 | Days since last update | +| β€” | Discussion Complexity | 1.5 | Thread count and distinct commenters | +| 6 | Alignment | 1 | Organizational signal | +| 7 | Freshness | 1 | Stale PRs less likely to merge soon | +| 9 | PR Size / Complexity | 1 | Ease of review | +| 10 | Community Contribution Flag | 0.5 | Flags community PRs β€” typically more expensive to drive | +| 13 | Review Velocity | 0.5 | Momentum signal | + +**Max possible raw score**: 20.5. Normalized to a 0–10 scale: `(rawScore / 20.5) Γ— 10`. + +### Formula + +``` +composite = Ξ£ (dimension_score Γ— weight) +``` + +Rank PRs by composite score descending. Ties are broken by: + +1. Higher CI Status score +2. Higher Maintainer Review score +3. Older `createdAt` (longer-waiting PRs first) + +--- + +## Edge Cases + +### Draft PRs + +Skip entirely in batch mode. In single-PR mode, note: *"This is a draft PR β€” +most dimensions don't apply until it's marked ready for review."* + +### Bot PRs + +`dotnet-maestro[bot]` (codeflow) PRs are excluded from batch by default. +`copilot-swe-agent` PRs are **included by default** β€” they are maintainer-initiated +and follow normal review workflows. Use `-ExcludeCopilot` to exclude them. + +### Flow PRs + +Missing packages β‰  infrastructure failure. Context drift is expected for +dependency-flow PRs. Deprioritize in ranking. + +### Backport PRs + +May target release branches. Check if the target branch has the test +infrastructure. Build Analysis may behave differently on non-main branches. diff --git a/.github/skills/pr-triage/references/runtime-signals.md b/.github/skills/pr-triage/references/runtime-signals.md new file mode 100644 index 00000000000000..71c3c3799a90b3 --- /dev/null +++ b/.github/skills/pr-triage/references/runtime-signals.md @@ -0,0 +1,143 @@ +# dotnet/runtime Signals Reference + +Runtime-specific signals, labels, area ownership, and automation behavior for PR triage. + +--- + +## Label Taxonomy + +| Label | Meaning | Applied by | Triage impact | +|-------|---------|-----------|--------------| +| `needs-author-action` | Reviewer requested changes, waiting on author | Maintainers, or auto when review has `Changes_requested` | ❌ Blocking β€” author must act | +| `needs-further-triage` | Author responded to `needs-author-action`, needs maintainer re-review | Auto (when author comments on non-untriaged issue) | ⚠️ Maintainer should re-engage | +| `untriaged` | PR hasn't been categorized by area yet | Auto on creation | ❌ Missing area routing | +| `no-recent-activity` | No activity for 14 days | Auto (resourceManagement.yml) | ⚠️ Will auto-close in 14 more days | +| `backlog-cleanup-candidate` | Issue inactive for 1644 days | Auto | Ignore for PR triage | +| `community-contribution` | PR from external contributor | Maintainers | Flagged for visibility so maintainers can prioritize | +| `area-*` (e.g., `area-System.Net`) | Component area label | Maintainers/auto | Used to find area owners | +| `api-ready-for-review` | Public API changes pending review | Author/maintainers | ⚠️ Needs API review before merge | +| `api-approved` | API review completed | API review board | βœ… API review done | +| `Known Build Error` | CI failure matched to known issue | Build Analysis automation | Used by ci-analysis skill | + +--- + +## Area Owners Lookup + +The file `docs/area-owners.md` contains a table mapping area labels to leads and owners. Format: + +``` +| Area | Lead | Owners (area experts to tag in PRs and issues) | Notes | +|------|------|------------------------------------------------|-------| +| area-System.Net.Http | @dotnet/ncl | @dotnet/ncl | | +``` + +### How to look up owners for a PR + +1. Get the PR's area label(s) from `gh pr view --json labels` +2. Parse the area-owners.md table (the script does this automatically at startup) +3. Match the area label to find the Lead and Owners columns +4. These are the people whose APPROVED review counts as "maintainer review" + +**Note**: The script parses `docs/area-owners.md` dynamically at startup to get +the full owner table (~138 areas). Do not rely on hardcoded owner lists. + +**Fallback**: If the PR has no area label, use `.github/CODEOWNERS` to match file paths to reviewers. + +--- + +## Approval Authority Levels + +Not all approvals carry equal weight for merge decisions. This reflects dotnet/runtime's +governance model where area owners have merge authority β€” it is not a judgment of review quality. + +| Level | Who | How to detect | Merge weight | +|-------|-----|--------------|--------------| +| **1. Area owner/lead** | Listed in Owners/Lead column of `docs/area-owners.md` for the PR's area label | Parse area-owners.md, match reviewer login against Owners column | Has merge authority for the area | +| **2. Community triager** | Trusted community members with triage permissions, listed in area-owners.md | Parse the "Community Triagers" section at the bottom of `docs/area-owners.md` | Strong signal β€” deeply familiar with the repo. Does not have merge authority but review is highly valued. | +| **3. Frequent contributor** | Has many merged PRs or commits in the touched area | `gh api "repos/dotnet/runtime/commits?author={login}&path={dir}&per_page=5"` β€” 3+ hits = frequent | Valuable domain expertise. Weight 1Γ— (standard). | +| **4. New contributor** | First-time or infrequent contributor to this area | No commits found in touched paths | Review appreciated β€” every contributor's feedback helps improve the PR. Weight 0.5Γ—. | + +### Known Community Triagers + +These are listed in the "Community Triagers" section of `docs/area-owners.md` and have triage permissions: + +@a74nh, @am11, @clamp03, @Clockwork-Muse, @filipnavara, @huoyaoyuan, @martincostello, @omajid, @Sergio0694, @shushanhf, @SingleAccretion, @teo-tsirpanis, @tmds, @vcsjones, @xoofx + +A Community Triager's APPROVED review is a stronger signal than a new contributor's approval. They are deeply familiar with the repo, its conventions, and its quality bar β€” but they are not area owners and their approval alone does not satisfy the "maintainer review" dimension. + +### Detecting reviewer level at runtime + +1. Get the PR's area label(s) +2. Parse `docs/area-owners.md` β€” match the area label row to get Owners +3. For each reviewer: check if their login appears in the Owners column (tier 1), the Community Triagers list (tier 2), or has recent commits in the area (tier 3). Otherwise tier 4. + +--- + +## resourceManagement.yml Automation + +The `.github/policies/resourceManagement.yml` file defines automated label management. + +### PR automation + +- PRs with `needs-author-action` + 14 days no activity β†’ `no-recent-activity` label added +- PRs with `no-recent-activity` + 14 more days β†’ auto-closed +- Draft PRs with 30 days no activity β†’ auto-closed +- Pushing to PR branch β†’ removes `needs-author-action` +- Author commenting β†’ removes `needs-author-action` +- Review with `changes_requested` by non-read-only user β†’ adds `needs-author-action` + +### What this means for triage + +- `needs-author-action` is a reliable "ball is in author's court" signal +- `no-recent-activity` means the PR is at risk of auto-closure +- Absence of `needs-author-action` does NOT mean feedback is addressed β€” check unresolved review threads too (the label system has gaps) + +--- + +## Community Contributions + +PRs with `community-contribution` label are flagged for visibility so maintainers can prioritize: + +- **Note in output** β€” the `is_community` flag lets maintainers filter and prioritize as they see fit +- **Be patient** with response times β€” community contributors have other commitments +- **Check author familiarity** β€” returning community contributors vs first-time contributors have different needs +- Timely feedback β€” even a quick "not right now" β€” respects contributors' time + +--- + +## Bot PRs + +| Bot | Label/Detection | Behavior | +|-----|----------------|----------| +| `dotnet-maestro[bot]` | `area-codeflow` label, author is `app/dotnet-maestro` | Dependency updates. Missing packages β‰  infrastructure. Different evaluation criteria. Exclude by default. | +| `copilot-swe-agent` | Author is `app/copilot-swe-agent` | Invoked by maintainers. Apply normal triage criteria. **Include by default.** Note agent authorship in output. | + +**Default**: Exclude `dotnet-maestro` PRs. Include `copilot-swe-agent` PRs (they are maintainer-initiated and follow normal review workflows). + +--- + +## Perf-Sensitive Paths + +Files in these paths should flag the perf-sensitivity dimension: + +- `src/libraries/System.Private.CoreLib/` β€” core runtime library +- `src/coreclr/jit/` β€” JIT compiler +- `src/coreclr/gc/` β€” garbage collector +- `src/coreclr/vm/` β€” VM runtime +- `src/libraries/System.Runtime/` β€” core types +- Any file with `Span`, `Memory`, `Buffer`, `Vector` in the path +- `src/libraries/System.Collections/` β€” hot collection types +- `src/libraries/System.Text.Json/` β€” high-perf serialization + +Check for @EgorBot benchmark comments to see if perf was already validated. + +--- + +## API Surface Paths + +Files matching these patterns indicate public API changes: + +- `src/libraries/*/ref/*.cs` β€” API reference assemblies +- `src/libraries/*/ref/*.csproj` β€” ref project files + +If changed, check for `api-ready-for-review` or `api-approved` labels. diff --git a/.github/skills/pr-triage/scripts/Get-PrTriageData.ps1 b/.github/skills/pr-triage/scripts/Get-PrTriageData.ps1 new file mode 100644 index 00000000000000..7add17258a527a --- /dev/null +++ b/.github/skills/pr-triage/scripts/Get-PrTriageData.ps1 @@ -0,0 +1,619 @@ +<# +.SYNOPSIS + Fetches and scores open PRs in dotnet/runtime for merge readiness. +.DESCRIPTION + Uses batched GraphQL to fetch reviews, review threads, and Build Analysis. + Outputs scored JSON for the AI skill to format and annotate. +.PARAMETER Label + Area label to filter by (e.g., "area-CodeGen-coreclr") +.PARAMETER Limit + Maximum PRs to return from gh pr list (default 500) +.PARAMETER Repo + Repository (default "dotnet/runtime") +.PARAMETER Maintainers + Optional list of usernames to treat as area owners (fallback when + area-owners.md is missing or has no match for a PR's labels). +.EXAMPLE + .\Get-PrTriageData.ps1 -Label "area-CodeGen-coreclr" +#> +[CmdletBinding()] +param( + [string]$Label, + [string]$Author, + [string]$Assignee, + [switch]$Community, + [int]$MinAge, + [int]$MaxAge, + [int]$UpdatedWithin, + [int]$MinApprovals, + [double]$MinScore, + [string]$HasLabel, + [string]$ExcludeLabel, + [switch]$IncludeDrafts, + [switch]$ExcludeCopilot, + [switch]$IncludeNeedsAuthor, + [switch]$IncludeStale, + [string]$MyActions, + [string]$NextAction, + [string]$PrNumber, + [int]$Top = 0, + [int]$Limit = 500, + [string]$Repo = "dotnet/runtime", + [string[]]$Maintainers = @(), + [switch]$OutputCsv +) + +$ErrorActionPreference = "Stop" + +# Handle comma-separated string from bash (e.g., "a,b,c" becomes one string element) +if ($Maintainers.Count -eq 1 -and $Maintainers[0] -match ',') { + $Maintainers = $Maintainers[0] -split ',' +} +$scriptStart = Get-Date + +# --- Area owners lookup (parsed from docs/area-owners.md) --- +$areaOwners = @{} +$repoParts = $Repo -split '/' +$areaOwnersUrl = "repos/$($repoParts[0])/$($repoParts[1])/contents/docs/area-owners.md" +try { + $areaOwnersMd = gh api -H "Accept: application/vnd.github.raw" $areaOwnersUrl 2>$null + foreach ($line in $areaOwnersMd -split "`n") { + if ($line -match '^\|\s*(area-\S+)\s*\|\s*@(\S+)\s*\|\s*(.+?)\s*\|') { + $areaName = $matches[1].Trim() + $lead = $matches[2].Trim().TrimEnd(',', ';') + $ownerField = $matches[3].Trim() + $people = @([regex]::Matches($ownerField, '@(\S+)') | ForEach-Object { $_.Groups[1].Value.TrimEnd(',', ';') } | + Where-Object { $_ -notmatch '^dotnet/' }) + if ($people.Count -eq 0) { $people = @($lead) } + $areaOwners[$areaName] = $people + } + } + Write-Verbose "Loaded $($areaOwners.Count) area owners from docs/area-owners.md" +} catch { + Write-Verbose "Warning: could not fetch area-owners.md, using empty owner table" +} + +$communityTriagers = @("a74nh","am11","clamp03","Clockwork-Muse","filipnavara", + "huoyaoyuan","martincostello","omajid","Sergio0694","shushanhf", + "SingleAccretion","teo-tsirpanis","tmds","vcsjones","xoofx") + +# --- Step 1: List PRs --- +Write-Verbose "Fetching PR list..." +$listArgs = @("pr","list","--repo",$Repo,"--state","open","--limit",$Limit, + "--json","number,title,author,labels,mergeable,isDraft,createdAt,updatedAt,changedFiles,additions,deletions,assignees") +if ($Label) { $listArgs += @("--label",$Label) } +if ($Author) { $listArgs += @("--author",$Author) } +if ($Assignee) { $listArgs += @("--assignee",$Assignee) } + +$prsRaw = & gh @listArgs | ConvertFrom-Json + +# --- Step 2: Quick-screen --- +$drafts = @($prsRaw | Where-Object { $_.isDraft }) +$bots = @($prsRaw | Where-Object { -not $_.isDraft -and $_.author.login -match "^(app/)?dotnet-maestro|^(app/)?github-actions" }) +$needsAuthor = @($prsRaw | Where-Object { -not $_.isDraft -and ($_.labels.name -contains "needs-author-action") }) +$stale = @($prsRaw | Where-Object { -not $_.isDraft -and ($_.labels.name -contains "no-recent-activity") }) +$candidates = @($prsRaw | Where-Object { + ($IncludeDrafts -or -not $_.isDraft) -and + $_.author.login -notmatch "^(app/)?dotnet-maestro|^(app/)?github-actions" -and + (-not $ExcludeCopilot -or $_.author.login -notmatch "copilot-swe-agent") -and + ($IncludeNeedsAuthor -or -not ($_.labels.name -contains "needs-author-action")) -and + ($IncludeStale -or -not ($_.labels.name -contains "no-recent-activity")) +}) + +# Apply additional label filters +if ($Community) { + $candidates = @($candidates | Where-Object { $_.labels.name -contains "community-contribution" }) +} +if ($HasLabel) { + $candidates = @($candidates | Where-Object { $_.labels.name -contains $HasLabel }) +} +if ($ExcludeLabel) { + $candidates = @($candidates | Where-Object { $_.labels.name -notcontains $ExcludeLabel }) +} + +# Apply age filters +$now = Get-Date +if ($MinAge -gt 0) { + $candidates = @($candidates | Where-Object { ($now - [DateTime]::Parse($_.createdAt)).TotalDays -ge $MinAge }) +} +if ($MaxAge -gt 0) { + $candidates = @($candidates | Where-Object { ($now - [DateTime]::Parse($_.createdAt)).TotalDays -le $MaxAge }) +} +if ($UpdatedWithin -gt 0) { + $candidates = @($candidates | Where-Object { ($now - [DateTime]::Parse($_.updatedAt)).TotalDays -le $UpdatedWithin }) +} + +# Single PR mode +if ($PrNumber) { + $candidates = @($candidates | Where-Object { $_.number -eq [long]$PrNumber }) + if ($candidates.Count -eq 0) { + # PR wasn't in filtered set - fetch it directly + $singlePr = & gh pr view $PrNumber --repo $Repo --json number,title,author,labels,mergeable,isDraft,createdAt,updatedAt,changedFiles,additions,deletions,assignees | ConvertFrom-Json + $candidates = @($singlePr) + } +} + +$excludedDrafts = if ($IncludeDrafts) { 0 } else { $drafts.Count } +$excludedBots = $bots.Count +Write-Verbose "Scanned $($prsRaw.Count) -> $($candidates.Count) candidates ($excludedDrafts drafts, $excludedBots bots, $($needsAuthor.Count) needs-author, $($stale.Count) stale excluded)" + +if ($candidates.Count -eq 0) { + Write-Verbose "No candidates to analyze." + @{ scanned = $prsRaw.Count; analyzed = 0; prs = @() } | ConvertTo-Json -Depth 5 + return +} + +# --- Step 3: Batched GraphQL (reviews, threads, Build Analysis, thread authors) --- +$fragment = 'number comments{totalCount} reviews(last:10){nodes{author{login}state commit{oid}}} reviewThreads(first:50){nodes{isResolved comments(first:5){nodes{author{login}}}}} commits(last:1){nodes{commit{oid statusCheckRollup{contexts(first:100){nodes{...on CheckRun{name conclusion status}}}}}}}' + +$graphqlData = @{} +$batches = [System.Collections.ArrayList]@() +$batch = [System.Collections.ArrayList]@() +foreach ($pr in $candidates) { + [void]$batch.Add($pr.number) + if ($batch.Count -eq 10) { + [void]$batches.Add([long[]]$batch.ToArray()) + $batch = [System.Collections.ArrayList]@() + } +} +if ($batch.Count -gt 0) { [void]$batches.Add([long[]]$batch.ToArray()) } + +$repoParts = $Repo -split '/' +$repoOwner = $repoParts[0] +$repoName = $repoParts[1] + +Write-Verbose "Fetching details in $($batches.Count) GraphQL batch(es)..." +foreach ($b in $batches) { + $parts = @() + for ($i = 0; $i -lt $b.Count; $i++) { + $parts += "pr$($i): pullRequest(number:$($b[$i])) { $fragment }" + } + $query = "{ repository(owner:`"$repoOwner`",name:`"$repoName`") { $($parts -join ' ') } }" + $result = gh api graphql -f query="$query" 2>&1 | ConvertFrom-Json + for ($i = 0; $i -lt $b.Count; $i++) { + $prData = $result.data.repository."pr$i" + if ($prData) { $graphqlData[$b[$i]] = $prData } + } +} + +# --- Step 4: Determine area owners for label --- +$owners = @() +if ($Label -and $areaOwners.ContainsKey($Label)) { + $owners = $areaOwners[$Label] +} +# Also try matching each PR's area labels +function Get-OwnersForPr($labelNames) { + foreach ($lbl in $labelNames) { + if ($areaOwners.ContainsKey($lbl)) { return $areaOwners[$lbl] } + } + return @() +} + +# --- Step 4b: Detect Copilot review errors (targeted query, avoids fetching body for all reviews) --- +$copilotErrorPRs = @{} +$prsWithCopilotReview = @($candidates | Where-Object { + $gql = $graphqlData[$_.number] + $gql -and ($gql.reviews.nodes | Where-Object { $_.author.login -eq "copilot-pull-request-reviewer" }) +}) +if ($prsWithCopilotReview.Count -gt 0) { + # Use larger batches since this is a lightweight fragment (only fetches copilot review bodies) + $copilotBatches = [System.Collections.ArrayList]@() + $cb = [System.Collections.ArrayList]@() + foreach ($pr in $prsWithCopilotReview) { + [void]$cb.Add($pr.number) + if ($cb.Count -eq 50) { [void]$copilotBatches.Add([long[]]$cb.ToArray()); $cb = [System.Collections.ArrayList]@() } + } + if ($cb.Count -gt 0) { [void]$copilotBatches.Add([long[]]$cb.ToArray()) } + Write-Verbose "Checking $($prsWithCopilotReview.Count) PR(s) for Copilot review errors in $($copilotBatches.Count) batch(es)..." + foreach ($b in $copilotBatches) { + $parts = @() + for ($i = 0; $i -lt $b.Count; $i++) { + $parts += "pr$($i): pullRequest(number:$($b[$i])) { number reviews(last:5) { nodes { author{login} body } } }" + } + $query = "{ repository(owner:`"$repoOwner`",name:`"$repoName`") { $($parts -join ' ') } }" + $result = gh api graphql -f query="$query" 2>&1 | ConvertFrom-Json + for ($i = 0; $i -lt $b.Count; $i++) { + $prData = $result.data.repository."pr$i" + if ($prData -and ($prData.reviews.nodes | Where-Object { + $_.author.login -eq "copilot-pull-request-reviewer" -and $_.body -match "Copilot encountered an error" + })) { + $copilotErrorPRs[$b[$i]] = $true + } + } + } + Write-Verbose "Found $($copilotErrorPRs.Count) PR(s) with Copilot review errors" +} + +# --- Step 5: Score each PR --- +$now = Get-Date +$results = @() + +foreach ($pr in $candidates) { + $n = $pr.number + $gql = $graphqlData[$n] + $labelNames = @($pr.labels | ForEach-Object { $_.name }) + + # Per-PR owners (use label-specific or fallback to filter-level, then -Maintainers) + $prOwners = Get-OwnersForPr $labelNames + if ($prOwners.Count -eq 0) { $prOwners = $owners } + if ($prOwners.Count -eq 0 -and $Maintainers.Count -gt 0) { $prOwners = $Maintainers } + + # For bot-authored PRs, find the human who triggered it (non-Copilot assignee) + $botTrigger = $null + if ($pr.author.login -match "^(app/)?copilot-swe-agent$") { + $botTrigger = $pr.assignees | Where-Object { $_.login -ne "Copilot" -and $_.login -ne "app/copilot-swe-agent" } | + Select-Object -First 1 -ExpandProperty login -ErrorAction SilentlyContinue + } + + # Extract Build Analysis + $checks = @() + $baConclusion = "ABSENT" + $headCommitOid = $null + if ($gql -and $gql.commits.nodes.Count -gt 0) { + $headCommitOid = $gql.commits.nodes[0].commit.oid + $rollup = $gql.commits.nodes[0].commit.statusCheckRollup + if ($rollup -and $rollup.contexts.nodes) { + $checks = @($rollup.contexts.nodes | Where-Object { $_.name }) + $baNode = $checks | Where-Object { $_.name -eq "Build Analysis" } | Select-Object -First 1 + if ($baNode) { + $baConclusion = if ($baNode.conclusion) { $baNode.conclusion } else { "IN_PROGRESS" } + } + } + } + + # Detect Copilot review errors (from pre-computed lookup) + $copilotReviewFailed = $copilotErrorPRs.ContainsKey($n) + + # Extract reviews (skip copilot reviewer) + $reviews = @() + if ($gql -and $gql.reviews.nodes) { + $reviews = @($gql.reviews.nodes | Where-Object { $_.author.login -ne "copilot-pull-request-reviewer" }) + } + + # Extract threads, commenters, discussion metrics + $unresolvedThreads = 0 + $totalThreads = 0 + $threadAuthors = @() + $allCommenters = @() + $prCommentCount = 0 + if ($gql) { + $prCommentCount = if ($gql.comments.totalCount) { $gql.comments.totalCount } else { 0 } + } + if ($gql -and $gql.reviewThreads.nodes) { + $totalThreads = $gql.reviewThreads.nodes.Count + $unresolved = @($gql.reviewThreads.nodes | Where-Object { -not $_.isResolved }) + $unresolvedThreads = $unresolved.Count + $threadAuthors = @($unresolved | ForEach-Object { + if ($_.comments.nodes.Count -gt 0) { $_.comments.nodes[0].author.login } + } | Where-Object { $_ } | Select-Object -Unique) + # All distinct commenters across all threads (resolved + unresolved) + $allCommenters = @($gql.reviewThreads.nodes | ForEach-Object { + $_.comments.nodes | ForEach-Object { $_.author.login } + } | Where-Object { $_ } | Select-Object -Unique) + } + $threadCommentSum = if ($gql -and $gql.reviewThreads.nodes) { + ($gql.reviewThreads.nodes | ForEach-Object { $_.comments.nodes.Count } | Measure-Object -Sum).Sum + } else { 0 } + $totalComments = $prCommentCount + $threadCommentSum + $distinctCommenters = $allCommenters.Count + + # Classify reviewers + $hasOwnerApproval = $false + $hasCurrentOwnerApproval = $false + $hasTriagerApproval = $false + $hasAnyApproval = $false + $hasStaleApproval = $false + $approvalCount = 0 + $reviewerLogins = @() + $approverLogins = @() + foreach ($rev in $reviews) { + $login = $rev.author.login + $reviewerLogins += $login + if ($rev.state -eq "APPROVED") { + $approvalCount++ + $hasAnyApproval = $true + $approverLogins += $login + # Check if approval is on the current head commit + $isStale = $headCommitOid -and $rev.commit -and $rev.commit.oid -and ($rev.commit.oid -ne $headCommitOid) + if ($isStale) { $hasStaleApproval = $true } + if ($prOwners -contains $login) { + $hasOwnerApproval = $true + if (-not $isStale) { $hasCurrentOwnerApproval = $true } + } + elseif ($communityTriagers -contains $login) { $hasTriagerApproval = $true } + } + } + $reviewerLogins = @($reviewerLogins | Select-Object -Unique) + $hasAnyReview = $reviews.Count -gt 0 + + # Labels + $isCommunity = $labelNames -contains "community-contribution" + $hasAreaLabel = ($labelNames | Where-Object { $_ -match "^area-" }).Count -gt 0 + $isUntriaged = $labelNames -contains "untriaged" + + # Dates + $updatedAt = [DateTime]::Parse($pr.updatedAt) + $createdAt = [DateTime]::Parse($pr.createdAt) + $daysSinceUpdate = ($now - $updatedAt).TotalDays + $ageInDays = ($now - $createdAt).TotalDays + + # Check counts + $passed = @($checks | Where-Object { $_.conclusion -eq "SUCCESS" }).Count + $failed = @($checks | Where-Object { $_.conclusion -eq "FAILURE" }).Count + $running = @($checks | Where-Object { $_.status -eq "IN_PROGRESS" -or $_.status -eq "QUEUED" }).Count + + # No Build Analysis check (non-runtime repos): infer CI from overall check results + if ($baConclusion -eq "ABSENT" -and $checks.Count -gt 0) { + if ($failed -gt 0) { $baConclusion = "FAILURE" } + elseif ($running -gt 0) { $baConclusion = "IN_PROGRESS" } + elseif ($passed -gt 0) { $baConclusion = "SUCCESS" } + } + + # --- DIMENSION SCORING --- + $ciScore = switch ($baConclusion) { "SUCCESS" { 1.0 } "ABSENT" { 0.5 } "IN_PROGRESS" { 0.5 } default { 0.0 } } + $stalenessScore = if ($daysSinceUpdate -le 3) { 1.0 } elseif ($daysSinceUpdate -le 14) { 0.5 } else { 0.0 } + $maintScore = if ($hasOwnerApproval) { 1.0 } elseif ($hasTriagerApproval) { 0.75 } elseif ($hasAnyReview) { 0.5 } else { 0.0 } + $hasNeedsAuthorAction = $labelNames -contains "needs-author-action" + $feedbackScore = if ($hasNeedsAuthorAction) { 0.0 } elseif ($unresolvedThreads -eq 0) { 1.0 } else { 0.5 } + $conflictScore = switch ($pr.mergeable) { "MERGEABLE" { 1.0 } "UNKNOWN" { 0.5 } "CONFLICTING" { 0.0 } default { 0.5 } } + $alignScore = if ($isUntriaged -or -not $hasAreaLabel) { 0.0 } else { 1.0 } + $freshScore = if ($daysSinceUpdate -le 14) { 1.0 } elseif ($daysSinceUpdate -le 30) { 0.5 } else { 0.0 } + $totalLines = $pr.additions + $pr.deletions + $sizeScore = if ($pr.changedFiles -le 5 -and $totalLines -le 200) { 1.0 } elseif ($pr.changedFiles -le 20 -and $totalLines -le 500) { 0.5 } else { 0.0 } + $communityScore = if ($isCommunity) { 0.5 } else { 1.0 } + # Stale approval: reduce approval score if approval is not on current commit + $approvalScore = if ($approvalCount -ge 2 -and $hasOwnerApproval) { 1.0 } + elseif ($hasOwnerApproval -or ($hasTriagerApproval -and $approvalCount -ge 2)) { 0.75 } + elseif ($hasTriagerApproval -or $approvalCount -ge 2) { 0.5 } + elseif ($approvalCount -ge 1) { 0.5 } + else { 0.0 } + if ($hasStaleApproval -and $approvalScore -gt 0) { $approvalScore = [Math]::Max(0, $approvalScore - 0.25) } + $velocityScore = if ($reviews.Count -eq 0) { if ($ageInDays -le 14) { 0.5 } else { 0.0 } } + elseif ($daysSinceUpdate -le 7) { 1.0 } elseif ($daysSinceUpdate -le 14) { 0.5 } else { 0.0 } + # Discussion complexity: many threads/commenters = harder to push forward + # Light (≀5 threads, ≀2 commenters) = 1.0, moderate = 0.5, heavy (>15 threads or >5 commenters) = 0.0 + $discussionScore = if ($totalThreads -le 5 -and $distinctCommenters -le 2) { 1.0 } + elseif ($totalThreads -le 15 -and $distinctCommenters -le 5) { 0.5 } + else { 0.0 } + + # Composite: weighted sum normalized to 0-10 scale + $rawMax = 20.0 + $rawScore = ($ciScore * 3) + ($conflictScore * 3) + ($maintScore * 3) + + ($feedbackScore * 2) + ($approvalScore * 2) + ($stalenessScore * 1.5) + + ($discussionScore * 1.5) + + ($alignScore * 1) + ($freshScore * 1) + ($sizeScore * 1) + + ($communityScore * 0.5) + ($velocityScore * 0.5) + $composite = [Math]::Round(($rawScore / $rawMax) * 10, 1) + + # --- WHO OWNS NEXT ACTION --- + # Identify 1-2 specific people responsible for the next step + $prNextAction = "" + $who = @() + + $authorLogin = $pr.author.login + + if ($pr.mergeable -eq "CONFLICTING") { + $prNextAction = "@$($authorLogin): resolve conflicts" + $who = @($authorLogin) + } + elseif ($baConclusion -eq "FAILURE") { + $prNextAction = "@$($authorLogin): fix CI failures" + $who = @($authorLogin) + } + elseif ($hasNeedsAuthorAction) { + $prNextAction = "@$($authorLogin): address feedback (needs-author-action)" + $who = @($authorLogin) + } + elseif ($unresolvedThreads -gt 0) { + $prNextAction = "@$($authorLogin): respond to $unresolvedThreads thread(s)" + $who = @($authorLogin) + # Note who's waiting on them (thread authors) + $waitingOn = @($threadAuthors | Where-Object { $_ -ne $authorLogin }) | Select-Object -First 2 + if ($waitingOn.Count -gt 0) { + $prNextAction += " from @$($waitingOn -join ', @')" + } + } + elseif (-not $hasAnyReview) { + $prNextAction = "Maintainer: review needed" + # Pick specific owners to tag: prefer owners who have reviewed similar PRs + if ($prOwners.Count -gt 0) { + $who = @($prOwners | Select-Object -First 2) + } else { + $who = @("area owner") + } + } + elseif ($daysSinceUpdate -gt 14) { + $prNextAction = "@$($authorLogin): merge main (stale $([int]$daysSinceUpdate)d)" + $who = @($authorLogin) + } + elseif ($hasOwnerApproval -and -not $hasCurrentOwnerApproval) { + $prNextAction = "Maintainer: re-review needed (approval on older commit)" + $staleOwners = @($approverLogins | Where-Object { $prOwners -contains $_ }) | Select-Object -First 2 + if ($staleOwners.Count -gt 0) { $who = $staleOwners } + elseif ($prOwners.Count -gt 0) { $who = @($prOwners | Select-Object -First 2) } + } + elseif ($ciScore -eq 1 -and $conflictScore -eq 1 -and $maintScore -ge 0.75 -and $feedbackScore -eq 1) { + $prNextAction = "Ready to merge" + # Who should click merge? The approving owner or area lead + if ($approverLogins.Count -gt 0) { + $who = @($approverLogins | Where-Object { $prOwners -contains $_ } | Select-Object -First 1) + if (-not $who -or $who.Count -eq 0) { $who = @($approverLogins | Select-Object -First 1) } + } elseif ($prOwners.Count -gt 0) { + $who = @($prOwners | Select-Object -First 1) + } + } + elseif (-not $hasOwnerApproval -and -not $hasTriagerApproval) { + $prNextAction = "Maintainer: review needed" + # Already have community review but need owner; pick owners not yet reviewing + $nonReviewingOwners = @($prOwners | Where-Object { $reviewerLogins -notcontains $_ }) | Select-Object -First 2 + if ($nonReviewingOwners.Count -gt 0) { $who = $nonReviewingOwners } + elseif ($prOwners.Count -gt 0) { $who = @($prOwners | Select-Object -First 2) } + } + elseif ($baConclusion -eq "IN_PROGRESS" -or $baConclusion -eq "ABSENT") { + $prNextAction = "Wait for CI" + $who = @($authorLogin) + } + else { + $prNextAction = "Maintainer: review/merge" + if ($prOwners.Count -gt 0) { $who = @($prOwners | Select-Object -First 2) } + } + + # For bot-authored PRs, substitute the human trigger person + if ($botTrigger -and $who.Count -gt 0 -and $who[0] -eq $pr.author.login) { + $who = @($botTrigger) + } + + # Append Copilot re-request suggestion if its review errored + if ($copilotReviewFailed) { + $prNextAction += "; rerequest Copilot review" + } + + $whoStr = if ($who.Count -gt 0) { "@" + ($who -join ", @") } else { "β€”" } + + # Blockers + $blockers = @() + if ($pr.mergeable -eq "CONFLICTING") { $blockers += "Conflicts" } + if ($baConclusion -eq "FAILURE") { $blockers += "CI fail ($failed failed)" } + if ($baConclusion -eq "IN_PROGRESS") { $blockers += "CI running" } + if ($baConclusion -eq "ABSENT") { $blockers += "No CI" } + if ($unresolvedThreads -gt 0) { $blockers += "$unresolvedThreads threads" } + if ($hasNeedsAuthorAction) { $blockers += "needs-author-action" } + if (-not $hasAnyReview) { $blockers += "No review" } + elseif (-not $hasOwnerApproval -and -not $hasTriagerApproval) { $blockers += "No owner approval" } + if ($daysSinceUpdate -gt 14) { $blockers += "Stale $([int]$daysSinceUpdate)d" } + if ($hasStaleApproval) { $blockers += "Approval not on latest commit" } + if ($copilotReviewFailed) { $blockers += "Copilot review errored" } + $blockersStr = if ($blockers.Count -gt 0) { $blockers -join ", " } else { "β€”" } + + # Why + $why = @() + $why += if ($ciScore -eq 1) { "βœ… CI passed" } elseif ($ciScore -eq 0) { "❌ CI failing" } else { "🟑 CI pending" } + if ($conflictScore -eq 0) { $why += "⚠️ has conflicts" } + if ($hasOwnerApproval) { $why += "πŸ‘ owner approved" } + elseif ($hasTriagerApproval) { $why += "πŸ‘ triager approved" } + elseif ($hasAnyApproval) { $why += "πŸ‘ community reviewed" } + elseif ($hasAnyReview) { $why += "πŸ‘€ reviewed, not approved" } + else { $why += "πŸ” no review yet" } + if ($hasStaleApproval) { $why += "⚠️ approval on older commit" } + if ($unresolvedThreads -gt 0) { $why += "πŸ’¬ $unresolvedThreads unresolved" } + if ($totalThreads -gt 15) { $why += "πŸ—¨οΈ busy ($totalThreads threads, $distinctCommenters people)" } + elseif ($totalThreads -gt 5) { $why += "πŸ—¨οΈ active ($totalThreads threads)" } + if ($isCommunity) { $why += "🌐 community" } + if ($sizeScore -eq 1) { $why += "πŸ“¦ small change" } + elseif ($sizeScore -eq 0) { $why += "πŸ“¦ large ($($pr.changedFiles) files, $($totalLines) lines)" } + if ($daysSinceUpdate -gt 14) { $why += "⏳ stale ($([int]$daysSinceUpdate)d)" } + if ($ageInDays -gt 90) { $why += "πŸ•°οΈ old ($([int]$ageInDays)d)" } + $whyStr = $why -join " Β· " + + $results += [PSCustomObject]@{ + number = $n + title = $pr.title.Substring(0, [Math]::Min(70, $pr.title.Length)) + author = $pr.author.login + score = $composite + ci = $baConclusion + ci_detail = "$passed/$failed/$running" + unresolved_threads = $unresolvedThreads + total_threads = $totalThreads + total_comments = $totalComments + distinct_commenters = $distinctCommenters + mergeable = $pr.mergeable + approval_count = $approvalCount + is_community = $isCommunity + age_days = [int]$ageInDays + days_since_update = [int]$daysSinceUpdate + changed_files = $pr.changedFiles + lines_changed = $totalLines + next_action = $prNextAction + who = $whoStr + blockers = $blockersStr + why = $whyStr + } +} + +# Sort by score descending +$results = $results | Sort-Object -Property score -Descending + +# --- Post-scoring filters --- +if ($MinApprovals -gt 0) { + $results = @($results | Where-Object { $_.approval_count -ge $MinApprovals }) +} +if ($MinScore -gt 0) { + $results = @($results | Where-Object { $_.score -ge $MinScore }) +} +if ($NextAction) { + # Filter by next-action type: "ready", "review", "author", "conflicts", "ci" + $pattern = switch ($NextAction.ToLower()) { + "ready" { "Ready to merge" } + "review" { "review needed" } + "author" { "^Author:" } + "conflicts" { "resolve conflicts" } + "ci" { "fix CI" } + default { $NextAction } + } + $results = @($results | Where-Object { $_.next_action -match $pattern }) +} +if ($MyActions) { + $me = $MyActions.TrimStart('@') + $results = @($results | Where-Object { + $_.who -match $me -or + ($_.author -eq $me -and $_.next_action -match "^Author:") + }) +} +$totalResults = $results.Count +if ($Top -gt 0) { + $results = @($results | Select-Object -First $Top) +} + +# --- Output JSON --- +$output = @{ + timestamp = $now.ToString("o") + repo = $Repo + filters = @{ + label = if ($Label) { $Label } else { $null } + author = if ($Author) { $Author } else { $null } + assignee = if ($Assignee) { $Assignee } else { $null } + community = [bool]$Community + min_age = $MinAge + max_age = $MaxAge + updated_within = $UpdatedWithin + min_approvals = $MinApprovals + min_score = $MinScore + next_action = if ($NextAction) { $NextAction } else { $null } + my_actions = if ($MyActions) { $MyActions } else { $null } + top = $Top + } + owners = $owners + scanned = $prsRaw.Count + analyzed = $candidates.Count + returned = $results.Count + total_after_filters = $totalResults + screened_out = @{ + drafts_count = $drafts.Count + drafts = @($drafts | Select-Object -First 10 | ForEach-Object { @{ number = $_.number; author = $_.author.login; title = $_.title.Substring(0, [Math]::Min(60, $_.title.Length)) } }) + bots = @($bots | ForEach-Object { @{ number = $_.number; author = $_.author.login } }) + needs_author_action = @($needsAuthor | ForEach-Object { @{ number = $_.number; author = $_.author.login } }) + stale = @($stale | ForEach-Object { @{ number = $_.number; author = $_.author.login } }) + } + quick_actions = @{ + ready_to_merge = @($results | Where-Object { $_.next_action -match "Ready to merge" }).Count + needs_maintainer_review = @($results | Where-Object { $_.next_action -match "review needed" }).Count + needs_author_action = @($results | Where-Object { $_.next_action -match "^Author:" }).Count + blocked_conflicts = @($results | Where-Object { $_.mergeable -eq "CONFLICTING" }).Count + } + prs = @($results) + elapsed_seconds = [Math]::Round(((Get-Date) - $scriptStart).TotalSeconds, 1) +} + +if ($OutputCsv) { + # Tab-separated output for easy SQL/spreadsheet import + $header = "number`ttitle`tauthor`tscore`tci`tci_detail`tunresolved_threads`ttotal_threads`ttotal_comments`tdistinct_commenters`tmergeable`tapproval_count`tis_community`tage_days`tdays_since_update`tchanged_files`tlines_changed`tnext_action`twho`tblockers`twhy" + $lines = @($header) + foreach ($r in $results) { + $t = ($r.title -replace "`t"," ").Substring(0, [Math]::Min(70, $r.title.Length)) + if ($t.Length -gt 0 -and $t[0] -in '=','+','-','@') { $t = "'$t" } + $lines += "$($r.number)`t$t`t$($r.author)`t$($r.score)`t$($r.ci)`t$($r.ci_detail)`t$($r.unresolved_threads)`t$($r.total_threads)`t$($r.total_comments)`t$($r.distinct_commenters)`t$($r.mergeable)`t$($r.approval_count)`t$(if ($r.is_community) {1} else {0})`t$($r.age_days)`t$($r.days_since_update)`t$($r.changed_files)`t$($r.lines_changed)`t$($r.next_action)`t$($r.who)`t$($r.blockers)`t$($r.why)" + } + $lines -join "`n" +} else { + $output | ConvertTo-Json -Depth 5 +}