Add ci-status skill for daily CI health dashboard#4094
Merged
Conversation
Creates a new skill that collects build status across main and recent release/* branches from both pipeline sources: - Public CI: SkiaSharp (Public) on xamarin/public (def 4) - Internal: SkiaSharp-Native (26493) → SkiaSharp (10789) → SkiaSharp-Tests (15756) on devdiv/DevDiv Shows the last N builds per pipeline per branch for a quick health overview. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
📦 Try the packages from this PRWarning Do not run these scripts without first reviewing the code in this PR. Step 1 — Download the packages bash / macOS / Linux: curl -fsSL https://raw.githubusercontent.com/mono/SkiaSharp/main/scripts/get-skiasharp-pr.sh | bash -s -- 4094PowerShell / Windows: iex "& { $(irm https://raw.githubusercontent.com/mono/SkiaSharp/main/scripts/get-skiasharp-pr.ps1) } 4094"Step 2 — Add the local NuGet source dotnet nuget add source ~/.skiasharp/hives/pr-4094/packages --name skiasharp-pr-4094More options
Or download manually from Azure Pipelines — look for the Remove the source when you're done: dotnet nuget remove source skiasharp-pr-4094 |
Contributor
|
📖 Documentation Preview The documentation for this PR has been deployed and is available at: 🔗 View Staging Site This preview will be updated automatically when you push new commits to this PR. This comment is automatically updated by the documentation staging workflow. |
For the most recent failed or partially-succeeded build on each pipeline, fetches the build timeline and displays errors and warnings inline. Shows up to 5 errors and 5 warnings per build with task name context. Use --no-issues to skip the timeline fetch for a faster overview. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove the 5-issue cap — now collects and displays all errors and warnings from the most recent run's timeline when it's failed or partially succeeded. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add --output flag to write a formatted markdown report with health summary table, per-branch build history, and collapsible error/warning sections - Remove exclusion of .x servicing branches (they are the main equivalent for their respective major.minor) - Refactor data collection into collect_data() for reuse by both renderers Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Enrich data collection: commit info, duration, trigger reason, pass rates, and green→red regression detection per pipeline - Add --json flag to output structured data for AI consumption - Cache ADO auth token for script lifetime (fewer subprocess calls) - Add get_build_changes() to fetch associated commits for failing builds - Rewrite SKILL.md with comprehensive AI analysis instructions: root cause clustering, classification decision tree, pipeline chain analysis, flake detection, cross-branch correlation, release risk assessment, and prioritized recommendations - Include regression detection table in markdown report - Show commit SHA and build duration in detailed tables Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Eval revealed the AI analysis step sometimes listed internal pipelines (Native/Managed/Tests) as independent failures instead of collapsing cascaded downstream failures to their upstream root cause. Make Step 2.3 mandatory with explicit per-branch cascade-vs-independent statements, and surface the chain verdict in the Step 3 presentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
3 eval scenarios (daily health check, full report with analysis, release readiness) used to benchmark the skill via skill-creator's evaluator. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a new
ci-statusskill that gives a broad, at-a-glance view of SkiaSharp CI health acrossmainand the recentrelease/*branches, then layers AI analysis on top to turn raw red/green into actionable diagnosis. Unlikerelease-status(which tracks a single release through its pipeline chain), this is a daily dashboard across all branches at once.Pipelines tracked
SkiaSharp (Public)onxamarin/public(definition 4) — push/PR tomain,release/*SkiaSharp-Native(26493) →SkiaSharp(10789) →SkiaSharp-Tests(15756) ondevdiv/DevDivWhat it does
scripts/ci-status.pyqueries each pipeline once and renders three views from a single data-collection pass:--output) — health matrix, per-branch commit info (SHA/author/message), build durations, pass-rate stats, and a green→red regression table--json) — consumed by the AI analysis stepIt auto-discovers the N most recent
release/*branches (including.xservicing branches), extracts errors/warnings from failed builds, caches the ADO token across requests, and pre-computes green→red regression transitions.SKILL.mddrives the AI layer: root-cause clustering by normalized error signature, a classification decision tree (infra / flake / regression / quota / chain blockage), mandatory pipeline-chain cascade analysis, flake detection, cross-branch correlation, release-risk assessment, and a capped top-5 prioritized action list — every claim tied to a real build ID and URL.Usage
Works well as a daily scheduled workflow.
Evaluation
Benchmarked with skill-creator's evaluator across three scenarios (daily health check, full report with analysis, release readiness). The eval surfaced one recurring miss — the analysis sometimes listed internal pipelines as independent failures instead of collapsing cascaded downstream failures to their upstream root cause — which is why Step 2.3 (chain cascade) is now mandatory and the chain verdict is surfaced in the presentation. Eval test cases are committed under
evals/so the skill stays benchmarkable.Triggers: "CI status", "build health", "is main green", "CI dashboard", "daily build status", "why is CI red", "is release/X shippable"