-
Notifications
You must be signed in to change notification settings - Fork 2k
Add Azure DevOps CI investigation instructions with az CLI preference #34335
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from 8 commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
53fdaf9
Add Azure DevOps CI investigation guidelines with az CLI preference
github-actions[bot] 431a2a3
Simplify az CLI guidance: drop instruction file, keep copilot-instruc…
github-actions[bot] 7902a61
Improve pr-build-status skill: zero-build triage, binlogtool, better …
github-actions[bot] bd92bd5
Rename skill: pr-build-status → azdo-build-investigator
github-actions[bot] 0ca160e
Clarify az login is optional for dnceng-public (public org)
github-actions[bot] 28c7f7b
Minimize token usage in copilot-instructions.md (~51% reduction)
github-actions[bot] 78d5ef2
Revert broad token reduction; keep only minimal az section changes
github-actions[bot] 8b27d08
skill: add az to tools, binlogtool reconstruct/doublewrites, error pa…
github-actions[bot] 99fb0b4
Apply multi-model review feedback to azdo-build-investigator skill
github-actions[bot] 31511b2
Fix binlog artifact download section - maui doesn't use 'binlog' arti…
github-actions[bot] 2ab2ed8
Fix Get-BuildErrors: detect test failures in passing Helix jobs
github-actions[bot] 0b377b1
azdo-build-investigator: add Get-BuildBinlogs.ps1 for Container artif…
github-actions[bot] f9d3fc9
azdo-build-investigator: fix Get-BuildBinlogs.ps1 (regex + OctetStrea…
github-actions[bot] File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,167 @@ | ||
| --- | ||
| name: azdo-build-investigator | ||
| description: "Investigate CI failures for dotnet/maui PRs — build errors, Helix test logs, and binlog analysis. Use when asked about failing checks, CI status, test failures, 'why is CI red', 'build failed', 'what's failing on PR', Helix failures, or device test failures." | ||
| metadata: | ||
| author: dotnet-maui | ||
| version: "1.2" | ||
| compatibility: Requires GitHub CLI (gh) authenticated with access to dotnet/maui repository. | ||
| --- | ||
|
|
||
| # PR Build Status Skill | ||
|
|
||
| Investigate CI failures for dotnet/maui PRs — build errors, Helix test logs, and binlog analysis. | ||
|
|
||
| ## Tools Required | ||
|
|
||
| This skill uses `bash` together with `pwsh` (PowerShell 7+) to run the PowerShell scripts. No file editing or other tools are required. | ||
|
|
||
| **If `gh` or `pwsh` is missing: stop immediately and tell the user to install the missing tool. Do NOT attempt to install it yourself.** | ||
|
|
||
| - `gh`: https://cli.github.com/ | ||
| - `pwsh`: https://aka.ms/install-powershell | ||
|
|
||
| Optional for binlog analysis (MSBuild failures): | ||
| - `az` (Azure CLI): `brew install azure-cli` / `winget install Microsoft.AzureCLI`, then `az extension add --name azure-devops` | ||
| - `binlogtool`: `dotnet tool install -g binlogtool` (https://www.nuget.org/packages/binlogtool) | ||
|
|
||
| ## When to Use | ||
|
|
||
| - User asks about CI/CD status for a PR | ||
| - User asks about failed checks or builds | ||
| - User asks "what's failing on PR #XXXXX" / "why is CI red" / "build failed" | ||
| - User wants to see test results | ||
| - **User asks about Helix failures (device tests, integration tests, etc.)** | ||
| - **User needs to debug why tests are failing on Helix infrastructure** | ||
| - **Text logs say "Build FAILED" with no detail — use binlog analysis** | ||
|
|
||
| ## Scripts | ||
|
|
||
| All scripts are in `.github/skills/azdo-build-investigator/scripts/` | ||
|
|
||
| ### 1. Get Build IDs for a PR | ||
| ```bash | ||
| pwsh .github/skills/azdo-build-investigator/scripts/Get-PrBuildIds.ps1 -PrNumber <PR_NUMBER> | ||
| ``` | ||
|
|
||
| ### 2. Get Build Status | ||
| ```bash | ||
| pwsh .github/skills/azdo-build-investigator/scripts/Get-BuildInfo.ps1 -BuildId <BUILD_ID> | ||
| # For failed jobs only: | ||
| pwsh .github/skills/azdo-build-investigator/scripts/Get-BuildInfo.ps1 -BuildId <BUILD_ID> -FailedOnly | ||
| ``` | ||
|
|
||
| ### 3. Get Build Errors and Test Failures | ||
| ```bash | ||
| # Get all errors (build errors + test failures) | ||
| pwsh .github/skills/azdo-build-investigator/scripts/Get-BuildErrors.ps1 -BuildId <BUILD_ID> | ||
|
|
||
| # Get only build/compilation errors | ||
| pwsh .github/skills/azdo-build-investigator/scripts/Get-BuildErrors.ps1 -BuildId <BUILD_ID> -ErrorsOnly | ||
|
|
||
| # Get only test failures | ||
| pwsh .github/skills/azdo-build-investigator/scripts/Get-BuildErrors.ps1 -BuildId <BUILD_ID> -TestsOnly | ||
| ``` | ||
|
|
||
| ### 4. Get Helix Console Logs | ||
| ```bash | ||
| # List all Helix work items and their status | ||
| pwsh .github/skills/azdo-build-investigator/scripts/Get-HelixLogs.ps1 -BuildId <BUILD_ID> | ||
|
|
||
| # Filter by platform | ||
| pwsh .github/skills/azdo-build-investigator/scripts/Get-HelixLogs.ps1 -BuildId <BUILD_ID> -Platform Windows | ||
|
|
||
| # Show console log content for failed work items | ||
| pwsh .github/skills/azdo-build-investigator/scripts/Get-HelixLogs.ps1 -BuildId <BUILD_ID> -ShowConsoleLog | ||
|
|
||
| # Filter by work item name and show more log lines | ||
| pwsh .github/skills/azdo-build-investigator/scripts/Get-HelixLogs.ps1 -BuildId <BUILD_ID> -WorkItem "*Lifecycle*" -ShowConsoleLog -TailLines 200 | ||
| ``` | ||
|
|
||
| ## Workflow | ||
|
|
||
| > **Focus on the first error chronologically — later errors usually cascade from the root cause.** | ||
|
|
||
| ### Standard Build Failures | ||
| 1. Get build IDs: `Get-PrBuildIds.ps1 -PrNumber XXXXX` | ||
| - If output shows ⚠️ with no build IDs, CI was not triggered — read the diagnostic message | ||
| 2. For each build, get status: `Get-BuildInfo.ps1 -BuildId YYYYY -FailedOnly` | ||
| 3. For failed builds, get errors: `Get-BuildErrors.ps1 -BuildId YYYYY` | ||
| 4. If errors say "Build FAILED" with no detail, check for binlog artifacts (see below) | ||
|
|
||
| ### Helix Test Failures | ||
| 1. Get build IDs: `Get-PrBuildIds.ps1 -PrNumber XXXXX` | ||
| 2. Find the build with Helix jobs (e.g., `maui-pr-devicetests`, `maui-integration-tests`) | ||
| 3. Get Helix logs: `Get-HelixLogs.ps1 -BuildId YYYYY -ShowConsoleLog` | ||
| 4. For specific platform: `Get-HelixLogs.ps1 -BuildId YYYYY -Platform Windows -ShowConsoleLog` | ||
|
|
||
| ### Binlog Analysis (MSBuild/XamlC/NuGet failures) | ||
| When text logs are inconclusive, `.binlog` artifacts contain the full MSBuild structured log. | ||
|
|
||
| **Requires `binlogtool`** (`dotnet tool install -g binlogtool`). If not installed, tell the user and stop. | ||
|
|
||
| ```bash | ||
| # Download the binlog artifact | ||
| az pipelines runs artifact download --run-id BUILD_ID --artifact-name "binlog" --path /tmp/maui-binlog \ | ||
| --org https://dev.azure.com/dnceng-public --project public | ||
|
|
||
| # Search for errors (broad first, then narrow) | ||
| binlogtool search "/tmp/maui-binlog/*.binlog" "error" | ||
| binlogtool search "/tmp/maui-binlog/*.binlog" "error CS" # C# compiler | ||
| binlogtool search "/tmp/maui-binlog/*.binlog" "error NU" # NuGet | ||
| binlogtool search "/tmp/maui-binlog/*.binlog" "XamlC" # XAML compiler | ||
| binlogtool search "/tmp/maui-binlog/*.binlog" "XA" # Android build errors | ||
|
|
||
| # Reconstruct full text log (useful when you need context around an error) | ||
| binlogtool reconstruct "/tmp/maui-binlog/*.binlog" > /tmp/maui-build.log | ||
|
|
||
| # Detect double-write errors (multiple tasks writing to the same output file) | ||
| binlogtool doublewrites "/tmp/maui-binlog/*.binlog" | ||
|
|
||
| # Clean up | ||
| Remove-Item -Recurse -Force /tmp/maui-binlog | ||
| ``` | ||
|
|
||
| ## Understanding Helix Logs | ||
|
|
||
| Helix is the .NET engineering infrastructure that runs tests across multiple platforms and device types. Tests that run on Helix include: | ||
| - **Device tests** - Run on real devices/emulators (iOS, Android, Windows, MacCatalyst) | ||
| - **Integration tests** - Run on various OS configurations | ||
| - **Other distributed tests** - Any test scenario that requires Helix infrastructure | ||
|
|
||
| When Helix tests fail: | ||
|
|
||
| 1. **Build stage** - Compiles and packages the test app/harness | ||
| 2. **Helix submission** - Sends the work items to Helix queues | ||
| 3. **Work item execution** - Helix runs the tests on target machines/devices | ||
| 4. **Console log** - Contains stdout/stderr from the test execution | ||
|
|
||
| The `Get-HelixLogs.ps1` script retrieves the console logs which show: | ||
| - Test execution output | ||
| - Any crashes or errors | ||
| - Infrastructure issues (timeouts, installation failures, etc.) | ||
|
|
||
| ## Common Build Error Patterns | ||
|
|
||
| | Pattern | Area | Notes | | ||
| |---------|------|-------| | ||
| | `error CS####` | C# compiler | Root cause; check file/line reference | | ||
| | `error NU1###` | NuGet restore | NU1301 = feed unreachable; NU11## = resolution failure | | ||
| | `XamlC` | XAML compiler | MAUI-specific; usually missing type or invalid binding | | ||
| | `##[error]` | ADO infrastructure | Pipeline-level error, not a build error | | ||
| | `System.TimeoutException` | Test infra | Infrastructure timeout; may be transient | | ||
| | `error MT####` | iOS/Mac linker | Linking failure; check build logs | | ||
| | `error BL####` | Build logic | MSBuild task failure | | ||
|
|
||
| ## Common Helix Failure Patterns | ||
|
|
||
| | Pattern in Console Log | Meaning | | ||
| |------------------------|---------| | ||
| | "XHarness timeout" | Test took too long, killed by infrastructure | | ||
| | "No test result files found" | Tests never ran or process crashed | | ||
| | "error MT..." or "error BL..." | Build/linking error (check build logs instead) | | ||
| | Exit code non-zero | Test failures or infrastructure issues | | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - `gh` (GitHub CLI) - authenticated | ||
| - `pwsh` (PowerShell 7+) | ||
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.