Add pr-triage skill for merge-readiness analysis #125005

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

danmoseley wants to merge 14 commits into dotnet:main from danmoseley:pr-triage-skill

.github/skills/ci-analysis/SKILL.md

Large diffs are not rendered by default.

.github/skills/ci-analysis/references/analysis-workflow.md

-Original file line number
+Diff line change
@@ -0,0 +1,39 @@
+    # Analysis Workflow (Steps 1–3)
+    After completing Step 0 (Gather Context — see SKILL.md), follow these steps.
+    ## Step 1: Run the Script
+    Run with `-ShowLogs` for detailed failure info. See [script-modes.md](script-modes.md) for parameter details.
+    ## Step 1b: Investigate with AzDO Tools
+    When the script output is insufficient (e.g., build timeline fetch fails), use AzDO MCP tools to query builds directly. **Match the org from the build URL to the correct AzDO tools** — see [azdo-helix-reference.md](azdo-helix-reference.md#azure-devops-organizations). PR builds are in `dnceng-public`; internal builds are in `dnceng`.
+    ## Step 2: Analyze Results
+. **Check Build Analysis** — If the Build Analysis GitHub check is **green**, all failures matched known issues and it's safe to retry. If it's **red**, some failures are unaccounted for — you must identify which failing jobs are covered by known issues and which are not. For 3+ failures, use SQL tracking to avoid missed matches (see [sql-tracking.md](sql-tracking.md)).
+. **Correlate with PR changes** — Same files failing = likely PR-related
+. **Compare with baseline** — If a test passes on the target branch but fails on the PR, compare Helix binlogs. See [binlog-comparison.md](binlog-comparison.md) — **delegate binlog download/extraction to subagents** to avoid burning context on mechanical work.
+. **Check build progression** — If the PR has multiple builds (multiple pushes), check whether earlier builds passed. A failure that appeared after a specific push narrows the investigation to those commits. See [build-progression-analysis.md](build-progression-analysis.md). Present findings as facts, not fix recommendations.
+. **Interpret patterns** (but don't jump to conclusions):
+       - Same error across many jobs → Real code issue
+       - Build Analysis flags a known issue → That *specific failure* is safe to retry (but others may not be)
+       - Failure is **not** in Build Analysis → Investigate further before assuming transient
+       - Device failures, Docker pulls, network timeouts → *Could* be infrastructure, but verify against the target branch first
+       - Test timeout but tests passed → Executor issue, not test failure
+. **Check for mismatch with user's question** — The script only reports builds for the current head SHA. If the user asks about a job, error, or cancellation that doesn't appear in the results, **ask** if they're referring to a prior build. Common triggers:
+       - User mentions a canceled job but `canceledJobNames` is empty
+       - User says "CI is failing" but the latest build is green
+       - User references a specific job name not in the current results
+       Offer to re-run with `-BuildId` if the user can provide the earlier build ID from AzDO.
+    ## Step 3: Verify Before Claiming
+    Before stating a failure's cause, verify your claim:
+    - **"Infrastructure failure"** → Did Build Analysis flag it? Does the same test pass on the target branch? If neither, don't call it infrastructure.
+    - **"Transient/flaky"** → Has it failed before? Is there a known issue? A single non-reproducing failure isn't enough to call it flaky.
+    - **"PR-related"** → Do the changed files actually relate to the failing test? Correlation in the script output is heuristic, not proof.
+    - **"Safe to retry"** → Are ALL failures accounted for (known issues or infrastructure), or are you ignoring some? Check the Build Analysis check status — if it's red, not all failures are matched. Map each failing job to a specific known issue before concluding "safe to retry."
+    - **"Not related to this PR"** → Have you checked if the test passes on the target branch? Don't assume — verify.

.github/skills/ci-analysis/references/azdo-helix-reference.md

-Original file line number
+Diff line change
@@ Expand Up @@
     ## Azure DevOps Organizations
-    **Public builds (default):**
-    - Organization: `dnceng-public`
-    - Project: `cbb18261-c48f-4abb-8651-8cdcb5474649`
+    Builds live in two Azure DevOps organizations. **Determine the org from the build URL**, then use AzDO tools for that org:
-    **Internal/private builds:**
-    - Organization: `dnceng`
-    - Project GUID: Varies by pipeline
+    | Organization | URL pattern | Project | When used |
+    |---|---|---|---|
+    | `dnceng-public` | `dev.azure.com/dnceng-public/...` | `public` (GUID: `cbb18261-c48f-4abb-8651-8cdcb5474649`) | PR validation builds (most common) |
+    | `dnceng` | `dev.azure.com/dnceng/...` | `internal` or varies | Official/internal builds, signed builds |
+    **How to pick the right tools:** Look at the build URL from `gh pr checks` output or the script's `[CI_ANALYSIS_SUMMARY]`. The URL contains the org name (e.g., `dev.azure.com/dnceng-public/...`). Search your available AzDO tools for ones matching that org name — there may be multiple sets of AzDO tools for different organizations. If a query returns null, you're likely using tools for the wrong org.
+    > ⚠️ **Common mistake:** Wrong org = null results. PR validation builds are almost always in `dnceng-public`. If you get null, check the org before trying other approaches.
     Override with:
     ```powershell
@@ Expand Down @@

.github/skills/ci-analysis/references/delegation-patterns.md

-Original file line number
+Diff line change
@@ Expand Up @@
     Steps:
 . Check if TRX test results are available for the work item. Parse them for pass/fail counts.
-. If no structured results, check for testResults.xml
-. Parse the XML for pass/fail counts on the <assembly> element
+       NOTE: hlx_test_results may error if no TRX files were uploaded (common for non-.NET-test work items like mobile device tests). If it errors, proceed to step 2.
+. If no structured results, search for testResults.xml using hlx_search_file(jobId, workItem, fileName="testResults.xml", pattern="total")
+. If found, parse the XML for pass/fail counts on the <assembly> element
+. If neither is available, check the console log for test summary lines using hlx_search_log(jobId, workItem, pattern="passed")
     Return JSON: { "jobName": "...", "hasResults": true, "passed": N, "failed": N }
     Or: { "jobName": "...", "hasResults": false, "reason": "no testResults.xml uploaded" }
@@ Expand Down @@

.github/skills/ci-analysis/references/failure-interpretation.md

-Original file line number
+Diff line change
@@ -0,0 +1,50 @@
+    # Interpreting CI Results
+    ## Result Categories
+    **Known Issues section**: Failures matching existing GitHub issues — these are tracked and being investigated.
+    **Build Analysis check status**: The "Build Analysis" GitHub check is **green** only when *every* failure is matched to a known issue. If it's **red**, at least one failure is unaccounted for — do NOT claim "all failures are known issues" just because some known issues were found. You must verify each failing job is covered by a specific known issue before calling it safe to retry.
+    **Canceled/timed-out jobs**: Jobs canceled due to earlier stage failures or AzDO timeouts. Dependency-canceled jobs don't need investigation. **Timeout-canceled jobs may have all-passing Helix results** — the "failure" is just the AzDO job wrapper timing out, not actual test failures. To verify: use `hlx_status` on each Helix job in the timed-out build (include passed work items). If all work items passed, the build effectively passed.
+    > ❌ **Don't dismiss timed-out builds.** A build marked "failed" due to a 3-hour AzDO timeout can have 100% passing Helix work items. Check before concluding it failed.
+    **PR Change Correlation**: Files changed by PR appearing in failures — likely PR-related.
+    **Build errors**: Compilation failures need code fixes.
+    **Helix failures**: Test failures on distributed infrastructure.
+    **Local test failures**: Some repos (e.g., dotnet/sdk) run tests directly on build agents. These can also match known issues — search for the test name with the "Known Build Error" label.
+    ## Per-Failure Details
+    `failedJobDetails` in JSON: Each failed job includes `errorCategory`, `errorSnippet`, and `helixWorkItems`. Use these for per-job classification instead of applying a single `recommendationHint` to all failures.
+    Error categories: `test-failure`, `build-error`, `test-timeout`, `crash` (exit codes 139/134/-4), `tests-passed-reporter-failed` (all tests passed but reporter crashed — genuinely infrastructure), `unclassified` (investigate manually).
+    > ⚠️ **`crash` does NOT always mean tests failed.** Exit code -4 often means the Helix work item wrapper timed out *after* tests completed. Always check `testResults.xml` before concluding a crash is a real failure. See [Recovering Results](#recovering-results-from-crashedcanceled-jobs) below.
+    > ⚠️ **Be cautious labeling failures as "infrastructure."** Only conclude infrastructure with strong evidence: Build Analysis match, identical failure on target branch, or confirmed outage. Exception: `tests-passed-reporter-failed` is genuinely infrastructure.
+    > ❌ **Missing packages on flow PRs ≠ infrastructure.** Flow PRs can cause builds to request *different* packages. Check *which* package and *why* before assuming feed delay.
+    ## Recovering Results from Crashed/Canceled Jobs
+    When an AzDO job is canceled (timeout) or Helix work items show `Crash` (exit code -4), the tests may have actually passed. Follow this procedure:
+. **Find the Helix job IDs** — Read the AzDO "Send to Helix" step log and search for lines containing `Sent Helix Job`. Extract the job GUIDs.
+. **Check Helix job status** — Get pass/fail summary for each job. Look at `failedCount` vs `passedCount`.
+. **For work items marked Crash/Failed** — Check if tests actually passed despite the crash. Try structured test results first (TRX parsing), then search for pass/fail counts in result files without downloading, then download as last resort:
+       - Parse the XML: `total`, `passed`, `failed` attributes on the `<assembly>` element
+       - If `failed=0` and `passed > 0`, the tests passed — the "crash" is the wrapper timing out after test completion
+. **Verdict**:
+       - All work items passed or crash-with-passing-results → **Tests effectively passed.** The failure is infrastructure (wrapper timeout).
+       - Some work items have `failed > 0` in testResults.xml → **Real test failures.** Investigate those specific tests.
+       - No testResults.xml uploaded → Tests may not have run at all. Check console logs for errors.
+    > This pattern is common with long-running test suites (e.g., WasmBuildTests) where tests complete but the Helix work item wrapper exceeds its timeout during result upload or cleanup.

.github/skills/ci-analysis/references/manual-investigation.md

-Original file line number
+Diff line change
@@ Expand Up @@
     $logContent | Select-String -Pattern "error|FAIL" -Context 2,5
     ```
+    ## Search Helix Logs and Artifacts Remotely
+    > 💡 **Prefer remote search over download.** `hlx_search_log` and `hlx_search_file` let you find errors in Helix console logs and uploaded files without downloading them first. Use these before falling back to `hlx_logs` (full log) or `hlx_download` (file download).
+    ```
+    # Search a work item's console log for error patterns
+    hlx_search_log(jobId, workItem, pattern="error", contextLines=3)
+    # Search an uploaded file (e.g., testResults.xml) for specific text
+    hlx_search_file(jobId, workItem, fileName="testResults.xml", pattern="Failed")
+    # Find which work items have specific file types
+    hlx_find_files(jobId, pattern="*.binlog")
+    ```
+    These tools return matching lines with context — much faster than downloading full logs and grepping locally. Use them for:
+    - Finding specific error messages across large console logs
+    - Searching test result files for failure patterns
+    - Locating crash dumps or binlogs across work items
     ## Query Helix APIs
-    > 💡 **Prefer MCP tools when available** — they handle most Helix queries without manual curl commands. Use the APIs below only as fallback.
+    > 💡 **Prefer MCP tools when available** — `hlx_search_log`, `hlx_search_file`, `hlx_status`, `hlx_logs`, `hlx_files`, etc. handle most Helix queries without manual curl commands. Use the APIs below only as fallback.
     ```bash
     # Get job details
@@ Expand Down @@

.github/skills/ci-analysis/references/recommendation-generation.md

-Original file line number
+Diff line change
@@ -0,0 +1,60 @@
+    # Generating Recommendations
+    After the script outputs the `[CI_ANALYSIS_SUMMARY]` JSON block, **you** synthesize recommendations. Do not parrot the JSON — reason over it.
+    ## Decision Logic
+    Read `recommendationHint` as a starting point, then layer in context:
+    | Hint | Action |
+    |------|--------|
+    | `BUILD_SUCCESSFUL` | No failures. Confirm CI is green. |
+    | `KNOWN_ISSUES_DETECTED` | Known tracked issues found — but this does NOT mean all failures are covered. Check the Build Analysis check status: if it's red, some failures are unmatched. Only recommend retry for failures that specifically match a known issue; investigate the rest. |
+    | `LIKELY_PR_RELATED` | Failures correlate with PR changes. Lead with "fix these before retrying" and list `correlatedFiles`. |
+    | `POSSIBLY_TRANSIENT` | Failures could not be automatically classified — does NOT mean they are transient. Use `failedJobDetails` to investigate each failure individually. |
+    | `REVIEW_REQUIRED` | Could not auto-determine cause. Review failures manually. |
+    | `MERGE_CONFLICTS` | PR has merge conflicts — CI won't run. Tell the user to resolve conflicts. Offer to analyze a previous build by ID. |
+    | `NO_BUILDS` | No AzDO builds found (CI not triggered). Offer to check if CI needs to be triggered or analyze a previous build. |
+    ## Layering Nuance
+    Then layer in nuance the heuristic can't capture:
+    - **Mixed signals**: Some failures match known issues AND some correlate with PR changes → separate them. Known issues = safe to retry; correlated = fix first.
+    - **Canceled jobs with recoverable results**: If `canceledJobNames` is non-empty, mention that canceled jobs may have passing Helix results (see [failure-interpretation.md](failure-interpretation.md) — Recovering Results).
+    - **Build still in progress**: If `lastBuildJobSummary.pending > 0`, note that more failures may appear.
+    - **Multiple builds**: If `builds` has >1 entry, `lastBuildJobSummary` reflects only the last build — use `totalFailedJobs` for the aggregate count.
+    - **BuildId mode**: `knownIssues` and `prCorrelation` won't be populated. Say "Build Analysis and PR correlation not available in BuildId mode."
+    ## How to Retry
+    - **AzDO builds**: Comment `/azp run {pipeline-name}` on the PR (e.g., `/azp run dotnet-sdk-public`)
+    - **All pipelines**: Comment `/azp run` to retry all failing pipelines
+    - **Helix work items**: Cannot be individually retried — must re-run the entire AzDO build
+    ## Tone and Output Format
+    Be direct. Lead with the most important finding. Structure your response as:
+. **Summary verdict** (1-2 sentences) — Is CI green? Failures PR-related? Known issues?
+. **Failure summary table** — narrow columns only (Job, Verdict, Issue). Keep job names short (drop redundant prefixes like `runtime-dev-innerloop`). Do NOT put error text or work item lists in the table — those go in the detail section below.
+. **Failure details** — one bullet per failure with error description, affected work items, and evidence
+. **Recommended actions** (numbered) — retry, fix, investigate. Include `/azp run` commands.
+    **Use markdown links** for PRs, issues, builds, and jobs so they're clickable: `[#121195](url)`, `[Build 1305302](url)`, `[job name](azdo-job-url)`. The script output and MCP tools provide the URLs — thread them through to your response.
+    Example layout for step 2+3:
+    ```
+    | # | Job | Verdict | Issue |
+    |---|-----|---------|-------|
+    | 1 | [browser-wasm linux Release WasmBuildTests](https://dev.azure.com/…) | Known flaky ✅ | [#121195](https://github.com/dotnet/runtime/issues/121195) |
+    | 2 | [linux-x64 Debug Mono Interpreter LibTests](https://dev.azure.com/…) | Known flaky ✅ | [#100800](https://github.com/dotnet/runtime/issues/100800) |
+    | 3 | [coreclr Pri0 Runtime Tests linux x64](https://dev.azure.com/…) | Known flaky ✅ | [#110173](https://github.com/dotnet/runtime/issues/110173) |
+    **Details:**
+    - **#1**: Playwright.TargetClosedException + dbus socket missing in WBT-NoWorkload
+    - **#2**: 8 work items (ComInterfaceGen, IntrinsicsInSPC, JSImportGen, …) — all exit code 139 (SIGSEGV), mono interpreter crashes
+    - **#3**: stackoverflowtester timeout in baseservices-exceptions — ASSERT: "Target stack has been corrupted"
+    ```
+    Synthesize from: JSON summary (structured facts) + human-readable output (details/logs) + Step 0 context (PR type, author intent).

.github/skills/ci-analysis/references/script-modes.md

-Original file line number
+Diff line change
@@ -0,0 +1,48 @@
+    # Script Modes and Parameters
+    ## Key Parameters
+    | Parameter | Description |
+    |-----------|-------------|
+    | `-PRNumber` | GitHub PR number to analyze |
+    | `-BuildId` | Azure DevOps build ID |
+    | `-ShowLogs` | Fetch and display Helix console logs |
+    | `-Repository` | Target repo (default: dotnet/runtime) |
+    | `-MaxJobs` | Max failed jobs to show (default: 5) |
+    | `-SearchMihuBot` | Search MihuBot for related issues |
+    ## Three Modes
+    The script operates in three distinct modes depending on what information you have:
+    | You have... | Use | What you get |
+    |-------------|-----|-------------|
+    | A GitHub PR number | `-PRNumber 12345` | Full analysis: all builds, failures, known issues, structured JSON summary |
+    | An AzDO build ID | `-BuildId 1276327` | Single build analysis: timeline, failures, Helix results |
+    | A Helix job ID (optionally a specific work item) | `-HelixJob "..." [-WorkItem "..."]` | Deep dive: list work items for the job, or with `-WorkItem`, focus on a single work item's console logs, artifacts, and test results |
+    > ❌ **Don't guess the mode.** If the user gives a PR URL, use `-PRNumber`. If they paste an AzDO build link, extract the build ID. If they reference a specific Helix job, use `-HelixJob`.
+    ## What the Script Does
+    ### PR Analysis Mode (`-PRNumber`)
+. Discovers AzDO builds associated with the PR (from GitHub check status; for full build history, query AzDO builds on `refs/pull/{PR}/merge` branch)
+. Fetches Build Analysis for known issues
+. Gets failed jobs from Azure DevOps timeline
+. **Separates canceled jobs from failed jobs** (canceled may be dependency-canceled or timeout-canceled)
+. Extracts Helix work item failures from each failed job
+. Fetches console logs (with `-ShowLogs`)
+. Searches for known issues with "Known Build Error" label
+. Correlates failures with PR file changes
+. **Emits structured summary** — `[CI_ANALYSIS_SUMMARY]` JSON block with all key facts for the agent to reason over
+    > **After the script runs**, you (the agent) generate recommendations. The script collects data; you synthesize the advice. See [recommendation-generation.md](recommendation-generation.md).
+    ### Build ID Mode (`-BuildId`)
+. Fetches the build timeline directly (skips PR discovery)
+. Performs steps 3–7 from PR Analysis Mode, but does **not** fetch Build Analysis known issues or correlate failures with PR file changes (those require a PR number). Still emits `[CI_ANALYSIS_SUMMARY]` JSON.
+    ### Helix Job Mode (`-HelixJob` [and optional `-WorkItem`])
+. With `-HelixJob` alone: enumerates work items for the job and summarizes their status
+. With `-HelixJob` and `-WorkItem`: queries the specific work item for status and artifacts
+. Fetches console logs and file listings, displays detailed failure information

Navigation Menu

Uh oh!

Add pr-triage skill for merge-readiness analysis #125005

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!