diff --git a/.github/docs/maui-ci-facts.md b/.github/docs/maui-ci-facts.md index 7cc03cf1f4f6..91b77a6082fc 100644 --- a/.github/docs/maui-ci-facts.md +++ b/.github/docs/maui-ci-facts.md @@ -354,17 +354,29 @@ on appearance alone: that it is the same failure as the name match, so it too is forced to `indeterminate`. A noisy/partially-present message still never inflates false reds. 2. **Job-level baseline match** — for a build break with no test name (crossgen/NativeAOT/ - linker/MSBuild), the same **leg** is also red on the most recent base build. Conversely, - a leg that is **red on the PR but green on base is PROOF the break is PR-caused** — this - is the strongest signal and a test-only diff cannot produce it. The automated lane now - **computes this in `Gather-TestFailureContext.ps1`** (per-failure `legBaselineResult` / - `legRegressedVsBase` / `legAlsoFailsOnBase` and a `deterministicAttribution` prior); the - interactive investigator does the same comparison by hand from the timelines. **Note the - asymmetry:** a leg being red on base (`legAlsoFailsOnBase`) is only **leg-level** - evidence — the leg can fail on base at a *different* test, so it does **not** on its own - prove *this* test is pre-existing. Only an **exact test+platform** base match - (`alsoFailsOnBaseline`, item 1) is strong enough to dismiss; a leg-only match is treated - as **indeterminate** (`Needs human investigation`), never dismissed. + linker/MSBuild), the same **leg** is also red on the base branch. Conversely, + a leg that is **red on the PR but green across several recent base builds is PROOF the + break is PR-caused** — this is the strongest signal and a test-only diff cannot produce + it. **Sample a few base builds, not one:** MAUI's UI suite is intermittently red on the + base branch, so a single green base build cannot tell a real regression from a flaky test + that merely happened to pass its one sampled base run. The automated lane computes the diff + over the **last few completed base builds of the PR's own base branch** (`main` for a + `main` PR, `net11.0` for a net11-targeting PR — `RegressionBaseBuilds`, default 5) and only + calls a leg `regressed-vs-base` when it was green on **at least `MinBaseGreenSamples`** + (default 2) of them and red on **none** — a deterministic build-error leg + (crossgen/NativeAOT/linker/MSBuild, which compiles or it doesn't) needs only one green base + build. A leg red on **some** sampled base builds and green on others is `flaky-on-base` + (never a regression); a leg green on base but on too few samples is + `succeeded-on-base-unconfirmed` (indeterminate, not a confident regression). The automated + lane **computes this in `Gather-TestFailureContext.ps1`** (per-failure `legBaselineResult` / + `legRegressedVsBase` / `legAlsoFailsOnBase`, the `baseSampleCount` / `baseGreenCount` / + `baseFailedCount` evidence, and a `deterministicAttribution` prior); the interactive + investigator does the same comparison by hand from the timelines. **Note the asymmetry:** a + leg being red on base (`legAlsoFailsOnBase`) is only **leg-level** evidence — the leg can + fail on base at a *different* test, so it does **not** on its own prove *this* test is + pre-existing. Only an **exact test+platform** base match (`alsoFailsOnBaseline`, item 1) is + strong enough to dismiss; a leg-only match is treated as **indeterminate** (`Needs human + investigation`), never dismissed. 3. **Known-issue match** — the failure message matches an open `Known Build Error` issue (the dotnet Build Analysis registry). Cite the issue number/link — but treat it as a **hint, not a dismissal**: a text match alone can shadow a real PR break with a broad @@ -408,15 +420,16 @@ device tests fail, so a green `maui-pr-devicetests` check is trusted only when a was positively observed all-zero; absent that, it caps to `Needs human investigation`), or when a failure can be attributed **neither** way — not a clean regression vs base, not pre-existing on base, not a -known issue (`gate.unattributedFailures > 0`; e.g. the base leg outcome was ambiguous, the -base build was missing/unreadable, or a device-test result fell outside the deterministic +known issue (`gate.unattributedFailures > 0`; e.g. the base leg was flaky, green on too few +base samples to confirm a regression (`succeeded-on-base-unconfirmed`), the base build was +missing/unreadable, or a device-test result fell outside the deterministic build-error class). A `pre-existing-on-base` or exact-match `known-issue` dismissal is also **refused** (downgraded to `indeterminate`) when the PR actually edits the failing test file (`scopeGuardTripped` — the PR may have changed the test so it now fails for a new reason that merely coincides with the base/known text) or when the PR and base failures of the same test have a known **reason conflict** (`baselineReasonConflict`). It is likewise **capped at `Not ready`** whenever a leg is red on the PR -but green on the same leg of the most recent base build (`gate.legsRegressedVsBase > 0` — the +but green across several recent base builds and red on none of them (`gate.legsRegressedVsBase > 0` — the computed job-level regression; a device-test BUILD break counts here, only device-test TEST results are excluded). A proven regression sets the ceiling to `Not ready` even when softer `Needs human investigation` reasons are also present — a definitive PR-introduced break is a diff --git a/.github/skills/review-test-failures/SKILL.md b/.github/skills/review-test-failures/SKILL.md index e0fe6b56de60..c42fa67db746 100644 --- a/.github/skills/review-test-failures/SKILL.md +++ b/.github/skills/review-test-failures/SKILL.md @@ -87,20 +87,25 @@ Key fields to use: investigation`**. A **SKIPPED** device-test check does not cap (tests did not run); a **RED** one is handled as an ordinary failing check. - `gate.legsRegressedVsBase` (+ `legsRegressedVsBaseNames[]`) — distinct failures that - are **red on the PR but GREEN on the same leg of the most recent completed base - build** (a deterministic, computed job-level regression). **Any value > 0 caps the - ceiling at `Not ready`** — a `Ready to merge` / `No failures found` verdict is then - forbidden. This is the comparison that catches build-job breaks (crossgen/R2R, + are **red on the PR but GREEN on the same leg across several recent completed base + builds and red on none of them** (a deterministic, computed job-level regression). + **Any value > 0 caps the ceiling at `Not ready`** — a `Ready to merge` / `No failures + found` verdict is then forbidden. Sampling several base builds (not one) is what + separates a real regression from a base-branch flake that merely happened to pass its + one sampled base run — except a **deterministic** build break (crossgen/NativeAOT/linker/ + MSBuild, which compiles or it doesn't), where a single green base build is proof enough. This is the comparison that catches build-job breaks (crossgen/R2R, NativeAOT) the test-level baseline cannot. A device-test BUILD break (`source = azdo-build-error`) IS counted here because it is deterministic; only device-test TEST results are excluded (XHarness exit-0 blind spot) — they are surfaced but never hard-capped. - `gate.unattributedFailures` (+ `unattributedFailureNames[]`) — distinct failures the deterministic prior could attribute **neither** way: not a clean regression vs base, not pre-existing on base, not a known issue (`deterministicAttribution = indeterminate`). - Causes: the base leg outcome was ambiguous (a duplicate/retried leaf name → - `inconclusive-on-base`), the base build was missing/unreadable, or a device-test TEST - result outside the build-error class. They are neither provably PR-caused nor dismissible - as pre-existing/known, so **any value > 0 caps the ceiling at `Needs human investigation`**. + Causes: the leg was flaky on base (red on some sampled base builds, green on others → + `flaky-on-base`), the leg was green on base but on too few samples to confirm a regression + (`succeeded-on-base-unconfirmed`), the base build was missing/unreadable, or a device-test + TEST result outside the build-error class. They are neither provably PR-caused nor + dismissible as pre-existing/known, so **any value > 0 caps the ceiling at `Needs human + investigation`**. - Evidence counts: `failuresAlsoOnBaseline`, `failuresMatchingKnownIssue`, `failuresRetriedStillFailing`, `baselineInconclusiveRows`. - `failures.unique[]` — distinct PR failures (deduped by test name + OS platform). This @@ -113,20 +118,28 @@ Key fields to use: recent base-branch build — scoped to the **same pipeline definition**, so a failure in one pipeline is never dismissed by a same-named failure that only occurred in another), - `legBaselineResult` / `legRegressedVsBase` / `legAlsoFailsOnBase` — the **computed - job-level baseline diff** for this failure's leg: `succeeded-on-base` + - `legRegressedVsBase = true` means the SAME leg passed on base and is now red on the - PR (strongest PR-caused signal); `failed-on-base` + `legAlsoFailsOnBase = true` means - the same **leg** was already red on base — but note this is only **leg-level** - corroboration, NOT proof that *this specific test* is pre-existing (the leg can fail - on base at a **different** test), so on its own it does **not** dismiss the failure; - `absent-on-base` means the leg name did not exist on the base build (indeterminate — - do not treat as a regression); `inconclusive-on-base` means the leg both failed and - passed on base (flaky on base / retried) — also indeterminate, never a regression. The - computed `regressed-vs-base` set is pre-filtered to stay trustworthy: a + job-level baseline diff** for this failure's leg, computed over the **last few completed + base builds** (not a single base build) so a base-branch flake is not mistaken for a + regression: `succeeded-on-base` + `legRegressedVsBase = true` means the SAME leg was + GREEN across several recent base builds and red on **none** of them, and is now red on + the PR (strongest PR-caused signal); `failed-on-base` + `legAlsoFailsOnBase = true` means + the same **leg** was already red on at least one sampled base build — but note this is + only **leg-level** corroboration, NOT proof that *this specific test* is pre-existing (the + leg can fail on base at a **different** test), so on its own it does **not** dismiss the + failure; `flaky-on-base` means the leg was red on some sampled base builds and green on + others (demonstrably flaky on base) — indeterminate, never a regression; + `succeeded-on-base-unconfirmed` means the leg was green on base but on too few samples + (fewer than `MinBaseGreenSamples`, e.g. only one readable base build) to rule out + flakiness — indeterminate, **not** a confident regression; `absent-on-base` means the leg + name did not exist on the sampled base builds (indeterminate — do not treat as a + regression). The computed `regressed-vs-base` set is pre-filtered to stay trustworthy: a provisioning/infrastructure failure (Android SDK `Failed to find package`, avdmanager, disk-full — environmental and nondeterministic) and any failure that was flaky on base in **another** leg are both held to `legRegressedVsBase = false` so they fall to - `indeterminate` rather than masquerading as a deterministic regression, + `indeterminate` rather than masquerading as a deterministic regression. Each failure also + carries `baseSampleCount` / `baseGreenCount` / `baseFailedCount` (how many recent base + builds were read, and on how many the leg was green vs red) as regression-confidence + evidence, - `deterministicAttribution` — a **computed prior** you MUST start from, one of `regressed-vs-base` (treat as **Likely PR-caused** unless you can cite why the base comparison is invalid, e.g. a known-flaky base leg), `pre-existing-on-base` (treat as @@ -153,9 +166,9 @@ Key fields to use: family**; `null` otherwise) — the `[ci-scan]` issues are the MAUI **CI Failure Scanner** (an agentic `ci-status-*` workflow) tracking `recurring` flakes, `regression`s, and `build break`s on the `main` / `net11.0` base branches across **many** builds — i.e. - multi-build base-branch history, strictly broader than the single most-recent base build - the leg diff can see. It is used in **one direction only**: when the leg diff computed a - single-base `regressed-vs-base` and ci-scan documents that exact test (`matchKind=test`) + multi-build base-branch history, strictly broader than the few recent base builds + the leg diff samples. It is used in **one direction only**: when the leg diff computed a + few-build `regressed-vs-base` and ci-scan documents that exact test (`matchKind=test`) or its whole leg (`matchKind=leg`, only for OneTimeSetUp/mass/env/build-break **leg-wide** issues) as failing on the base branch, the regression is **demoted to `indeterminate`** (NHI) and `ciScanDemoted=true` is set. This is a **false-RED reduction only** — a ci-scan @@ -221,7 +234,7 @@ Classify each distinct failure as exactly one of: | Verdict | Use when | | --- | --- | -| `Likely PR-caused` | The failure directly references changed files, changed tests, changed APIs, affected platform code, or a newly added/modified test; or it only appears in a path/platform this PR changes and does **not** match a baseline failure or a known issue. **A `deterministicAttribution = regressed-vs-base` failure** (its leg is red on the PR but GREEN on the same leg of the most recent base build) is **computed, decisive** PR-caused evidence — default to this verdict unless you can cite why the base comparison is invalid (e.g. a known-flaky base leg). A `retriedStillFailing = true` failure in the PR's area is **stronger** PR-caused evidence (CI retried it and it still failed — it is not a one-off flake). | +| `Likely PR-caused` | The failure directly references changed files, changed tests, changed APIs, affected platform code, or a newly added/modified test; or it only appears in a path/platform this PR changes and does **not** match a baseline failure or a known issue. **A `deterministicAttribution = regressed-vs-base` failure** (its leg is red on the PR but GREEN across several recent base builds and red on none of them) is **computed, decisive** PR-caused evidence — default to this verdict unless you can cite why the base comparison is invalid (e.g. a known-flaky base leg, or `baseGreenCount` is small). A `retriedStillFailing = true` failure in the PR's area is **stronger** PR-caused evidence (CI retried it and it still failed — it is not a one-off flake). | | `Likely unrelated` | Evidence points to infrastructure, missing baselines, known flaky tests, unrelated platforms/areas, base/main failures, or the **exact same test+platform also fails on the baseline** (`alsoFailsOnBaseline = true` / `deterministicAttribution = pre-existing-on-base` — the only base signal strong enough to dismiss on its own). A known issue **corroborated by an exact base match** (`deterministicAttribution = known-issue`) is also unrelated — cite the issue number/link. **Caution:** `legAlsoFailsOnBase = true` *alone* (the leg was red on base but this exact test was not matched), a `matchesKnownIssue` hit whose `deterministicAttribution` is **`indeterminate`** (text match not corroborated by an **exact** base match), or a `baselineReasonConflict = true` failure (exact name match but a different known failure reason), is **NOT** sufficient to dismiss — those are `Needs human investigation`, not `Likely unrelated`. | | `Needs human investigation` | Evidence is mixed: the failure overlaps the PR area or platform but no direct causal link is clear, or the data suggests multiple plausible causes. | | `Insufficient data` | Build records, test results, or logs are missing/inaccessible/expired, or there is not enough evidence to make a responsible claim. | @@ -340,7 +353,7 @@ top-level `
` block. The `Overall` badge shows the **merge-readiness** v | Failure | Verdict | On base? | Evidence | | --- | --- | --- | --- | -| [check/test/build] | [Likely PR-caused | Likely unrelated | Needs human investigation | Insufficient data] | [yes/no — use the leg diff: `regressed` when `legRegressedVsBase`, `also-red` when `legAlsoFailsOnBase`, else the test-level `alsoFailsOnBaseline`] | [specific evidence — lead with `deterministicAttribution` when it is `regressed-vs-base`/`pre-existing-on-base`, cite a known-issue link when `matchesKnownIssue` is set, cite the `[ci-scan]` issue + occurrence count when `matchesCiScan` is set (and note it as `Needs human investigation` when `ciScanDemoted` — a single-base regression contradicted by multi-build base-branch history), note `retried still failing` when true, link build/test IDs] | +| [check/test/build] | [Likely PR-caused | Likely unrelated | Needs human investigation | Insufficient data] | [yes/no — use the leg diff: `regressed` when `legRegressedVsBase`, `also-red` when `legAlsoFailsOnBase`, else the test-level `alsoFailsOnBaseline`] | [specific evidence — lead with `deterministicAttribution` when it is `regressed-vs-base`/`pre-existing-on-base`, cite the base sampling (`baseGreenCount` green / `baseFailedCount` red of `baseSampleCount` base builds) for a regression, cite a known-issue link when `matchesKnownIssue` is set, cite the `[ci-scan]` issue + occurrence count when `matchesCiScan` is set (and note it as `Needs human investigation` when `ciScanDemoted` — a few-build regression contradicted by multi-build base-branch history), note `retried still failing` when true, link build/test IDs] | ### Recommended action diff --git a/.github/skills/review-test-failures/scripts/Gather-TestFailureContext.Tests.ps1 b/.github/skills/review-test-failures/scripts/Gather-TestFailureContext.Tests.ps1 index f52a3aa6a14c..4996e202c4b5 100644 --- a/.github/skills/review-test-failures/scripts/Gather-TestFailureContext.Tests.ps1 +++ b/.github/skills/review-test-failures/scripts/Gather-TestFailureContext.Tests.ps1 @@ -40,7 +40,13 @@ BeforeAll { 'Get-HelixWorkItemCounts', 'Get-XUnitFailures', 'Get-ConsoleFailureReason', - 'New-DeviceWorkItemFailureRecords' + 'New-DeviceWorkItemFailureRecords', + 'Get-AggregatedBaseLegMap', + 'Get-PlatformFromText', + 'Get-ErrorFingerprint', + 'Get-BuildErrorSignature', + 'Test-IsTransientBuildErrorCode', + 'Get-BuildErrorsFromLog' )) { $function = $ast.Find({ $args[0] -is [System.Management.Automation.Language.FunctionDefinitionAst] -and @@ -474,3 +480,141 @@ Describe 'New-DeviceWorkItemFailureRecords (classify ONE failed work item — ne $recs[0]['source'] | Should -Be 'helix-trx' } } + +Describe 'Get-AggregatedBaseLegMap (multi-build base leg diff — network-free via pre-seeded cache)' { + # The aggregator only calls Get-TimelineRecordResultMap on a CACHE MISS, so pre-seeding $Cache with + # entries keyed "org|project|buildId" is a fully network-free seam: each case supplies its own base + # single-build leg maps and asserts the green/red tallies that decide whether a PR leg is a clean + # 'regressed-vs-base' or a base flake. A cache MISS would call the (here-undefined) + # Get-TimelineRecordResultMap and throw, so a clean return also proves no network was attempted. + # Single-build cache entry shape mirrors Get-TimelineRecordResultMap: { accessible; records: + # normName -> { name; hasFailed; hasSucceeded } }. + + It 'counts a leg GREEN across all sampled base builds (all-green -> greenCount=N, failedCount=0)' { + $cache = @{ + 'o|p|101' = [ordered]@{ accessible = $true; records = @{ 'leg a' = [ordered]@{ name = 'Leg A'; hasFailed = $false; hasSucceeded = $true } } } + 'o|p|102' = [ordered]@{ accessible = $true; records = @{ 'leg a' = [ordered]@{ name = 'Leg A'; hasFailed = $false; hasSucceeded = $true } } } + } + $agg = Get-AggregatedBaseLegMap -Org 'o' -Project 'p' -BaseBuilds @([ordered]@{ id = 101 }, [ordered]@{ id = 102 }) -Cache $cache + $agg.accessible | Should -BeTrue + $agg.sampledBuilds | Should -Be 2 + $agg.records['leg a'].greenCount | Should -Be 2 + $agg.records['leg a'].failedCount | Should -Be 0 + } + + It 'tallies a leg red on some base builds and green on others (green-plus-red)' { + $cache = @{ + 'o|p|201' = [ordered]@{ accessible = $true; records = @{ 'leg a' = [ordered]@{ name = 'Leg A'; hasFailed = $true; hasSucceeded = $false } } } + 'o|p|202' = [ordered]@{ accessible = $true; records = @{ 'leg a' = [ordered]@{ name = 'Leg A'; hasFailed = $false; hasSucceeded = $true } } } + 'o|p|203' = [ordered]@{ accessible = $true; records = @{ 'leg a' = [ordered]@{ name = 'Leg A'; hasFailed = $false; hasSucceeded = $true } } } + } + $agg = Get-AggregatedBaseLegMap -Org 'o' -Project 'p' -BaseBuilds @([ordered]@{ id = 201 }, [ordered]@{ id = 202 }, [ordered]@{ id = 203 }) -Cache $cache + $agg.sampledBuilds | Should -Be 3 + $agg.records['leg a'].greenCount | Should -Be 2 + $agg.records['leg a'].failedCount | Should -Be 1 + } + + It 'counts a retry-then-pass base build as RED for that build (hasFailed wins over hasSucceeded)' { + # A base build where the leg failed one attempt but a retry later passed still carries a + # base-branch flake -> it must count RED, never GREEN, so it cannot mask a base flake and let a + # matching PR-red occurrence be read as a clean regression. + $cache = @{ 'o|p|301' = [ordered]@{ accessible = $true; records = @{ 'leg a' = [ordered]@{ name = 'Leg A'; hasFailed = $true; hasSucceeded = $true } } } } + $agg = Get-AggregatedBaseLegMap -Org 'o' -Project 'p' -BaseBuilds @([ordered]@{ id = 301 }) -Cache $cache + $agg.records['leg a'].failedCount | Should -Be 1 + $agg.records['leg a'].greenCount | Should -Be 0 + } + + It 'skips an INACCESSIBLE base build (not counted, not sampled)' { + $cache = @{ + 'o|p|401' = [ordered]@{ accessible = $false; records = @{ 'leg a' = [ordered]@{ name = 'Leg A'; hasFailed = $false; hasSucceeded = $true } } } + 'o|p|402' = [ordered]@{ accessible = $true; records = @{ 'leg a' = [ordered]@{ name = 'Leg A'; hasFailed = $false; hasSucceeded = $true } } } + } + $agg = Get-AggregatedBaseLegMap -Org 'o' -Project 'p' -BaseBuilds @([ordered]@{ id = 401 }, [ordered]@{ id = 402 }) -Cache $cache + $agg.sampledBuilds | Should -Be 1 + $agg.records['leg a'].greenCount | Should -Be 1 + $agg.baseBuildIds.Count | Should -Be 1 + $agg.baseBuildIds[0] | Should -Be 402 + } + + It 'returns accessible=$false when NO base build was readable' { + $cache = @{ 'o|p|501' = [ordered]@{ accessible = $false; records = @{ 'leg a' = [ordered]@{ name = 'Leg A'; hasFailed = $false; hasSucceeded = $true } } } } + $agg = Get-AggregatedBaseLegMap -Org 'o' -Project 'p' -BaseBuilds @([ordered]@{ id = 501 }) -Cache $cache + $agg.accessible | Should -BeFalse + $agg.sampledBuilds | Should -Be 0 + } + + It 'REUSES the shared cache across calls (a base id fetched once serves later PR builds — network-free)' { + # The outer loop shares ONE $baseRecordMapCache across PR builds; a base id read for one PR + # build must be reused for the next without a second fetch (a cache MISS would call the + # undefined Get-TimelineRecordResultMap and throw). + $cache = @{ 'o|p|601' = [ordered]@{ accessible = $true; records = @{ 'leg a' = [ordered]@{ name = 'Leg A'; hasFailed = $false; hasSucceeded = $true } } } } + $agg1 = Get-AggregatedBaseLegMap -Org 'o' -Project 'p' -BaseBuilds @([ordered]@{ id = 601 }) -Cache $cache + $agg2 = Get-AggregatedBaseLegMap -Org 'o' -Project 'p' -BaseBuilds @([ordered]@{ id = 601 }) -Cache $cache + $agg1.records['leg a'].greenCount | Should -Be 1 + $agg2.records['leg a'].greenCount | Should -Be 1 + $cache.Keys.Count | Should -Be 1 + } + + It 'ignores base builds with a non-positive id' { + $cache = @{ 'o|p|701' = [ordered]@{ accessible = $true; records = @{ 'leg a' = [ordered]@{ name = 'Leg A'; hasFailed = $false; hasSucceeded = $true } } } } + $agg = Get-AggregatedBaseLegMap -Org 'o' -Project 'p' -BaseBuilds @([ordered]@{ id = 0 }, [ordered]@{ id = 701 }) -Cache $cache + $agg.sampledBuilds | Should -Be 1 + $agg.baseBuildIds.Count | Should -Be 1 + $agg.baseBuildIds[0] | Should -Be 701 + } +} + +Describe 'Test-IsTransientBuildErrorCode (transient infra vs deterministic toolchain boundary)' { + It 'classifies restore/network + file-lock codes as transient' { + Test-IsTransientBuildErrorCode -Signature 'NU1301' | Should -BeTrue + Test-IsTransientBuildErrorCode -Signature 'MSB3021' | Should -BeTrue + Test-IsTransientBuildErrorCode -Signature 'MSB3027' | Should -BeTrue + } + + It 'classifies deterministic compiler/toolchain codes as NOT transient' { + Test-IsTransientBuildErrorCode -Signature 'CS0246' | Should -BeFalse # C# compile error + Test-IsTransientBuildErrorCode -Signature 'NU1101' | Should -BeFalse # package not found (deterministic) + Test-IsTransientBuildErrorCode -Signature 'MSB4018' | Should -BeFalse # task failed unexpectedly (deterministic) + Test-IsTransientBuildErrorCode -Signature 'Failed to load assembly' | Should -BeFalse + Test-IsTransientBuildErrorCode -Signature 'CrossGen/R2R' | Should -BeFalse + } + + It 'treats an empty/whitespace signature as NOT transient' { + Test-IsTransientBuildErrorCode -Signature '' | Should -BeFalse + Test-IsTransientBuildErrorCode -Signature ' ' | Should -BeFalse + } +} + +Describe 'Get-BuildErrorsFromLog (deterministicBuildError boundary — one-green-base shortcut gate)' { + It 'flags a transient NuGet restore/network code (NU1301) as NON-deterministic' { + $r = @(Get-BuildErrorsFromLog -Lines @('##[error]error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/x/index.json') -LogId 10 -RecordName 'Build_iOS') + $r.Count | Should -Be 1 + $r[0].deterministicBuildError | Should -BeFalse + } + + It 'flags transient MSBuild file-lock codes (MSB3021 / MSB3027) as NON-deterministic' { + $r1 = @(Get-BuildErrorsFromLog -Lines @('error MSB3021: Unable to copy file "a.dll" to "b.dll". The process cannot access the file because it is being used by another process.') -LogId 11 -RecordName 'Build_Android') + $r1[0].deterministicBuildError | Should -BeFalse + $r2 = @(Get-BuildErrorsFromLog -Lines @('error MSB3027: Could not copy "a.dll" to "b.dll". Exceeded retry count of 10. Failed. The file is locked by: "dotnet".') -LogId 12 -RecordName 'Build_Android') + $r2[0].deterministicBuildError | Should -BeFalse + } + + It 'keeps a genuine deterministic compile break (CS0246) as deterministic' { + $r = @(Get-BuildErrorsFromLog -Lines @('Foo.cs(12,5): error CS0246: The type or namespace name ''Bar'' could not be found') -LogId 13 -RecordName 'Build_Windows') + $r.Count | Should -Be 1 + $r[0].deterministicBuildError | Should -BeTrue + } + + It 'does NOT blanket-exclude the NU/MSB prefixes (NU1101, MSB4018 stay deterministic)' { + $rNu = @(Get-BuildErrorsFromLog -Lines @('error NU1101: Unable to find package Foo. No packages exist with this id.') -LogId 14 -RecordName 'Build_iOS') + $rNu[0].deterministicBuildError | Should -BeTrue + $rMsb = @(Get-BuildErrorsFromLog -Lines @('error MSB4018: The "GenerateResource" task failed unexpectedly.') -LogId 15 -RecordName 'Build_iOS') + $rMsb[0].deterministicBuildError | Should -BeTrue + } + + It 'keeps a native crash NON-deterministic (unchanged behavior)' { + $r = @(Get-BuildErrorsFromLog -Lines @('Process terminated. Segmentation fault (core dumped)') -LogId 16 -RecordName 'Run_iOS') + $r.Count | Should -Be 1 + $r[0].deterministicBuildError | Should -BeFalse + } +} diff --git a/.github/skills/review-test-failures/scripts/Gather-TestFailureContext.ps1 b/.github/skills/review-test-failures/scripts/Gather-TestFailureContext.ps1 index 70330251755e..3db016ec78b9 100644 --- a/.github/skills/review-test-failures/scripts/Gather-TestFailureContext.ps1 +++ b/.github/skills/review-test-failures/scripts/Gather-TestFailureContext.ps1 @@ -22,6 +22,20 @@ .PARAMETER LookbackBuilds Number of recent base-branch builds to include for each AzDO definition. +.PARAMETER RegressionBaseBuilds + Number of recent completed base-branch builds to SAMPLE for the job-level + regression diff. A failing leg is only called a clean `regressed-vs-base` + regression when it is GREEN across several recent base builds and red on none + of them -- a single base sample cannot tell a real regression from a base + branch flake (a UI test that merely happened to pass its one sampled base run). + +.PARAMETER MinBaseGreenSamples + Minimum number of recent base builds on which a (non-deterministic) failing + leg must be GREEN -- with zero base failures in the sampled window -- before it + is asserted as a clean `regressed-vs-base` regression. Deterministic build-error + legs (crossgen/NativeAOT/linker/MSBuild) compile or they don't, so one green + base build is proof enough for those. + .PARAMETER OutputDirectory Root directory for output. A PR-number subdirectory is created below it. @@ -44,6 +58,12 @@ param( [Parameter(Mandatory = $false)] [int]$LookbackBuilds = 5, + [Parameter(Mandatory = $false)] + [int]$RegressionBaseBuilds = 5, + + [Parameter(Mandatory = $false)] + [int]$MinBaseGreenSamples = 2, + [Parameter(Mandatory = $false)] [int]$BaselineBuildsPerDefinition = 1, @@ -608,6 +628,27 @@ function Get-BuildErrorSignature { return $null } +function Test-IsTransientBuildErrorCode { + # Returns $true when a coded build-error signature (from Get-BuildErrorSignature) denotes a + # TRANSIENT restore/network or file-lock break rather than a deterministic compile-or-it-doesn't + # toolchain error. This is the boundary for the single-green-base regression shortcut: a + # deterministic compiler/linker/SDK break reproduces build-to-build, so ONE green base sample is + # proof it is PR-introduced; a transient infra break does NOT reproduce, so one lucky green base + # sample must not flip it to 'regressed-vs-base' and hard-cap the verdict to 'Not ready' -- those + # stay subject to MinBaseGreenSamples (multi-sample flake protection), exactly like a crash/OOM. + # + # Conservative, explicit blocklist (unknown coded errors default to deterministic, since genuine + # compile breaks are the overwhelming majority and DO reproduce). Only whole-code, unambiguously + # non-deterministic infra codes are listed -- NOT whole prefixes: most NUxxxx (NU1101 package not + # found, NU1605 downgrade) and most MSBxxxx are deterministic and MUST keep the shortcut. + param([string]$Signature) + if ([string]::IsNullOrWhiteSpace($Signature)) { return $false } + # NU1301 : "Unable to load the service index for source" -- feed unreachable / restore network. + # MSB3021: "Unable to copy file ... The process cannot access the file" -- build-output file lock. + # MSB3027: "Could not copy ... exceeded retry count ... file is locked" -- build-output file lock. + return ($Signature -in @('NU1301', 'MSB3021', 'MSB3027')) +} + function Get-FailureReasonSignature { # Extracts a STABLE, low-cardinality "why did it fail" token from a failure's message(s) so two # failures of the SAME test that fail for DIFFERENT reasons (e.g. a PR-introduced @@ -1169,10 +1210,21 @@ function Get-BuildErrorsFromLog { $message = ([string]$line).Trim() $platform = Get-PlatformFromText -Text "$RecordName $message" + # Crash/OOM signatures (native-crash, test-host-crash, unhandled-exception, no-space-left) are + # NONDETERMINISTIC: a green base sample does not prove the PR caused them. So are TRANSIENT + # restore/network + file-lock coded breaks (NU1301, MSB3021, MSB3027 -- see + # Test-IsTransientBuildErrorCode): they emit a coded 'error XXnnnn:' line but vary run-to-run. + # Only deterministic compile/toolchain breaks (coded MSBuild/C#/SDK/linker errors, + # 'Failed to load assembly', CrossGen/R2R) reproduce build-to-build, so only THOSE may later + # take the single-green-base regression shortcut; crashes and transient infra breaks stay + # subject to MinBaseGreenSamples. + $isDeterministicBuildBreak = ($signature -notin @('native-crash', 'unhandled-exception', 'test-host-crash', 'no-space-left')) -and + (-not (Test-IsTransientBuildErrorCode -Signature $signature)) $failures.Add([ordered]@{ testName = "$RecordName - $signature" platform = $platform source = "azdo-build-error" + deterministicBuildError = $isDeterministicBuildBreak logId = $LogId recordName = $RecordName errorFingerprint = $fingerprint @@ -1191,6 +1243,7 @@ function Get-BuildErrorsFromLog { testName = "$RecordName - build error" platform = $platform source = "azdo-build-error" + deterministicBuildError = $false logId = $LogId recordName = $RecordName errorFingerprint = Get-ErrorFingerprint -Text $fallbackErrorLine @@ -1352,8 +1405,9 @@ function Get-RecentBaseBuilds { function Get-TimelineRecordResultMap { # Builds a deterministic map of leg/record name -> pass/fail outcome for ONE build's - # timeline. Used for the job-level baseline diff: a build leg that is red on the PR but - # GREEN on the most recent base build is the strongest possible PR-caused signal, and + # timeline. This is the single-build primitive that Get-AggregatedBaseLegMap samples across + # several base builds for the job-level baseline diff: a build leg that is red on the PR but + # GREEN across recent base builds is the strongest possible PR-caused signal, and # unlike a test-name match it works for build-job breaks (crossgen/NativeAOT/linker) # that carry no test name. Fully mechanical -- no LLM judgment. param( @@ -1391,6 +1445,60 @@ function Get-TimelineRecordResultMap { return $result } +function Get-AggregatedBaseLegMap { + # Builds a MULTI-BUILD deterministic leg map for the job-level regression diff. For each + # normalized leg/record name it counts, across the last N completed base-branch builds, + # how many ran the leg purely GREEN vs how many ran it RED (failed/partiallySucceeded). + # + # Why not a single build: the one-build diff (Get-TimelineRecordResultMap on the tip base + # build alone) cannot tell a real regression from a base-branch flake. MAUI's UI suite is + # intermittently red on the base branch, so a flaky test is green on SOME base builds and + # red on others; comparing against the ONE most-recent base build that happened to be + # green mislabels it "regressed vs base / Likely PR-caused". Sampling several base builds + # removes that false positive: a leg is a clean regression only when it is green across + # MULTIPLE recent base builds and red on NONE of them. Fully mechanical -- no LLM judgment. + param( + [string]$Org, + [string]$Project, + [object[]]$BaseBuilds, # completed base builds, newest-first + [hashtable]$Cache # memoized single-build maps keyed "org|project|buildId" + ) + + $agg = @{} + $sampled = 0 + $ids = New-Object System.Collections.Generic.List[int] + foreach ($base in @($BaseBuilds)) { + $bid = [int]$base.id + if ($bid -le 0) { continue } + $key = "$Org|$Project|$bid" + if (-not $Cache.ContainsKey($key)) { + $Cache[$key] = Get-TimelineRecordResultMap -Org $Org -Project $Project -BuildId $bid + } + $single = $Cache[$key] + if (-not $single.accessible) { continue } + $sampled++ + $ids.Add($bid) + foreach ($norm in @($single.records.Keys)) { + $rec = $single.records[$norm] + if (-not $agg.ContainsKey($norm)) { + $agg[$norm] = [ordered]@{ name = $rec.name; greenCount = 0; failedCount = 0 } + } + # Per base build, classify the leg once: a leg that failed even one attempt that + # build counts as RED for that build (a retry that later passed does not clear a + # base-branch flake); a leg that only ever succeeded that build counts as GREEN. + if ($rec.hasFailed) { $agg[$norm].failedCount++ } + elseif ($rec.hasSucceeded) { $agg[$norm].greenCount++ } + } + } + + return [ordered]@{ + accessible = ($sampled -gt 0) + records = $agg + sampledBuilds = $sampled + baseBuildIds = @($ids.ToArray()) + } +} + function Get-KnownBuildIssues { # Loads the repo's open "Known Build Error" issues (the dotnet Build Analysis # known-issues registry). Each such issue body carries one or more ```json blocks @@ -1502,11 +1610,11 @@ function Get-CiScanIssues { # Loads the repo's open '[ci-scan]' issues -- the MAUI-specific CI Failure Scanner # registry (an agentic 'ci-status-*' workflow) that tracks RECURRING flakes, # REGRESSIONS, and BUILD BREAKS on the main / net11.0 base branches across MANY builds. - # Unlike the single-base-build leg diff (which sees only the ONE most recent base build), - # this is multi-build, branch-scoped base-branch history. We parse each issue into a + # Unlike the few-build leg diff (which samples only the last few base builds), this is + # deeper multi-build, branch-scoped base-branch history. We parse each issue into a # matcher (branch family + class + the set of test-name tokens it documents + affected # leg tokens + occurrence text) so a PR failure that matches a documented base-branch - # failure can be DEMOTED off a single-base 'regressed-vs-base' claim to 'indeterminate' + # failure can be DEMOTED off a few-build 'regressed-vs-base' claim to 'indeterminate' # (NHI). This is a false-RED reduction only: a ci-scan match can never turn a red check # green (it is an LLM-generated hint, never a dismissal-to-green signal). param([string]$Repository) @@ -1808,7 +1916,7 @@ else { # ci-scan registry: MAUI's multi-build base-branch failure history (recurring flakes, # regressions, build breaks on main / net11.0), scoped to the PR's base branch family. Used -# ONLY to demote a single-base 'regressed-vs-base' claim to NHI when the same failure is +# ONLY to demote a few-build 'regressed-vs-base' claim to NHI when the same failure is # documented on the base branch -- a false-RED reduction, never a dismissal-to-green. $prBaseBranchFamily = Get-BranchFamily -Branch ([string]$pr.baseRefName) Write-Host "Loading ci-scan registry ('ci-scan' issues) for branch family '$prBaseBranchFamily'..." @@ -2595,7 +2703,12 @@ foreach ($buildRef in $buildRefsById.Values) { if ($build.definition -and $build.definition.id) { $definitionId = [int]$build.definition.id } - $buildSummary.recentBaseBuilds = @(Get-RecentBaseBuilds -Org $buildRef.org -Project $buildRef.project -DefinitionId $definitionId -BaseBranch $pr.baseRefName -Top $LookbackBuilds) + # Fetch enough recent base builds to satisfy BOTH the baseline lookback AND the regression-diff + # sampling window: RegressionBaseBuilds only trims an already-fetched list, so fetching just + # $LookbackBuilds would silently cap the leg diff (e.g. -RegressionBaseBuilds 10 with the default + # -LookbackBuilds 5 could sample at most 5 and miss a base failure in an omitted build). + $baseFetchTop = [Math]::Max($LookbackBuilds, $RegressionBaseBuilds) + $buildSummary.recentBaseBuilds = @(Get-RecentBaseBuilds -Org $buildRef.org -Project $buildRef.project -DefinitionId $definitionId -BaseBranch $pr.baseRefName -Top $baseFetchTop) $builds.Add($buildSummary) } @@ -2615,10 +2728,11 @@ $baselineRaw = New-Object System.Collections.Generic.List[object] $baselineSummary = New-Object System.Collections.Generic.List[object] $baselineInspected = @{} # Deterministic job-level baseline diff state. $prBuildToBaseMap maps each inspected PR -# build id -> the most recent completed base build's per-leg pass/fail map, so each PR -# failed leg can be compared to the SAME leg on base in code (no LLM judgment). -# $baseRecordMapCache memoizes the base timeline fetch so PR builds that share a base -# build (e.g. retried runs of one pipeline) don't re-fetch it. +# build id -> an AGGREGATED per-leg pass/fail count map over the last few completed base +# builds, so each PR failed leg can be compared to the SAME leg across several base builds +# in code (no LLM judgment). $baseRecordMapCache memoizes each base build's single-build +# timeline fetch so PR builds that share a base window (e.g. retried runs of one pipeline) +# don't re-fetch it. $prBuildToBaseMap = @{} $baseRecordMapCache = @{} @@ -2641,24 +2755,32 @@ if ($BaselineBuildsPerDefinition -gt 0) { $mostRecent = $completed[0] # --- Deterministic job-level baseline diff (fetch base leg map) --- - # Fetch the most recent completed base build's per-leg pass/fail map ONCE so each PR - # failed leg can later be compared to the SAME leg on base. This runs BEFORE the - # succeeded-base early-return below precisely because the strongest regression signal - # (leg red on PR, GREEN on a fully-succeeded base) lives in that branch. Unlike a - # test-name match this also catches build-job breaks (crossgen/NativeAOT/linker) that - # carry no test name -- the class of break that previously slipped through. + # Sample the last few completed base builds' per-leg pass/fail maps so each PR failed + # leg can later be compared to the SAME leg across MULTIPLE base builds. This runs + # BEFORE the succeeded-base early-return below precisely because the strongest + # regression signal (leg red on PR, GREEN across recent base builds) lives in that + # branch. Unlike a test-name match this also catches build-job breaks + # (crossgen/NativeAOT/linker) that carry no test name -- the class of break that + # previously slipped through. $isDeviceTestsDef = $defName -like '*devicetest*' - $baseMapKey = "$($build.org)|$($build.project)|$($mostRecent.id)" - if (-not $baseRecordMapCache.ContainsKey($baseMapKey)) { - $baseRecordMapCache[$baseMapKey] = Get-TimelineRecordResultMap -Org $build.org -Project $build.project -BuildId ([int]$mostRecent.id) - } - $baseMap = $baseRecordMapCache[$baseMapKey] - if ($baseMap.accessible) { + # Sample the last few completed base builds (not just the tip) so the job-level diff + # can tell a real regression from a base-branch flake. $baseRecordMapCache memoizes + # each build's single-build timeline map so PR builds sharing a base window don't + # re-fetch it (and so the tip build fetched here is reused below). + # Exclude canceled base builds from the regression-diff window: a canceled build has an + # incomplete timeline, so its missing legs count as neither green nor red and can make a leg + # look "all-green on base" -> a false regressed-vs-base signal. (The most-recent-tip baseline + # above is unaffected: it only early-returns on result -eq 'succeeded'.) + $legSampleBuilds = @($completed | Where-Object { $_.result -ne 'canceled' } | Select-Object -First $RegressionBaseBuilds) + $baseAgg = Get-AggregatedBaseLegMap -Org $build.org -Project $build.project -BaseBuilds $legSampleBuilds -Cache $baseRecordMapCache + if ($baseAgg.accessible) { $prBuildToBaseMap[[string]$build.id] = [ordered]@{ baseBuildId = [int]$mostRecent.id baseBuildResult = [string]$mostRecent.result isDeviceTests = $isDeviceTestsDef - records = $baseMap.records + records = $baseAgg.records + sampledBaseBuilds = [int]$baseAgg.sampledBuilds + baseBuildIds = @($baseAgg.baseBuildIds) } } @@ -2879,20 +3001,24 @@ foreach ($failure in $dedupedFailures) { # multi-build base-branch failure registry for THIS PR's base branch family? A hit means the # failure is documented to occur on the base branch independent of this PR (recurring flake, # known regression, or env instability across many builds) -- stronger and broader than the - # single most-recent base build the leg diff can see. Surfaced for the human on every failure; - # used below ONLY to demote a single-base 'regressed-vs-base' to NHI (never to dismiss-to-green). + # few recent base builds the leg diff samples. Surfaced for the human on every failure; + # used below ONLY to demote a few-build 'regressed-vs-base' to NHI (never to dismiss-to-green). $ciScanLegNames = @(@($failure.occurrences) | ForEach-Object { [string](Get-ObjectValue -Object $_ -Names @("recordName")) } | Where-Object { $_ }) $failure['matchesCiScan'] = Test-CiScanMatch -Matchers $ciScanIssues.matchers -TestName ([string]$failure.testName) -LegNames $ciScanLegNames -BranchFamily $prBaseBranchFamily # Deterministic job-level baseline diff: compare each occurrence's failing leg to the - # SAME leg on the most recent completed base build. base green + PR red = regressed - # (PR-caused); base also red = pre-existing. Device-test legs are surfaced via - # legBaselineResult but never set legRegressedVsBase (XHarness exit-0 blind spot), so - # the hard ceiling cap only fires on trustworthy maui-pr build results. + # SAME leg across the last few completed base builds. Green across several base builds + + # red on none + PR red = clean regression (PR-caused); red on any sampled base build = + # pre-existing / base-branch flake. Device-test legs are surfaced via legBaselineResult + # but never set legRegressedVsBase (XHarness exit-0 blind spot), so the hard ceiling cap + # only fires on trustworthy maui-pr build results. $legBaselineResult = $null $legRegressed = $false $legAlsoFails = $false $legInconclusive = $false + $baseSampleCount = 0 + $baseGreenCount = 0 + $baseFailedCount = 0 # Provisioning/infrastructure failures (Android SDK package fetch, emulator/avdmanager # setup, disk exhaustion) are ENVIRONMENTAL and nondeterministic -- the same flake lands on # different legs run-to-run. They must never establish a *deterministic* regression vs base: @@ -2911,55 +3037,94 @@ foreach ($failure in $dedupedFailures) { continue } $baseInfo = $prBuildToBaseMap[$occBuildId] + $sampled = [int]$baseInfo.sampledBaseBuilds + if ($sampled -gt $baseSampleCount) { $baseSampleCount = $sampled } $norm = ($occRecord -replace '\s+', ' ').Trim().ToLowerInvariant() if (-not $baseInfo.records.ContainsKey($norm)) { if (-not $legBaselineResult) { $legBaselineResult = 'absent-on-base' } continue } $baseRec = $baseInfo.records[$norm] - if ($baseRec.hasFailed -and $baseRec.hasSucceeded) { - # The same normalized leg name both FAILED and SUCCEEDED on base -- duplicate - # records share a leaf name across stages/jobs, or the leg was retried. The base - # outcome for THIS leg is ambiguous, so assert neither pre-existing nor regressed - # and let the failure fall through to 'indeterminate' (the gate then refuses a - # green verdict on it via unattributedFailures). This prevents both a false - # pre-existing subtraction (false green) and a false regression (false red). - if (-not $legBaselineResult) { $legBaselineResult = 'inconclusive-on-base' } - $legInconclusive = $true - } - elseif ($baseRec.hasFailed) { - # Same leg already failed on base -> pre-existing, regardless of any green attempt. + $green = [int]$baseRec.greenCount + $failed = [int]$baseRec.failedCount + if ($green -gt $baseGreenCount) { $baseGreenCount = $green } + if ($failed -gt $baseFailedCount) { $baseFailedCount = $failed } + if ($failed -ge 1) { + # The SAME leg failed on the base branch in at least one of the sampled base builds + # -> the break is present on base independent of this PR (pre-existing, or a + # base-branch flake), never a clean PR-introduced regression. This subsumes the old + # single-build "also-red" and, critically, catches the intermittently-failing UI + # legs a one-build diff would have mislabelled "regressed vs base". $legAlsoFails = $true $legBaselineResult = 'failed-on-base' + if ($green -ge 1) { + # Failed on some sampled base builds, green on others -> demonstrably FLAKY ON + # BASE. Mark inconclusive so a matching PR-red occurrence is not read as a + # regression even if another leg happened to see it green (the cross-leg veto + # below then suppresses any competing clean-regression claim). + $legInconclusive = $true + $legBaselineResult = 'flaky-on-base' + } } - elseif ($baseRec.hasSucceeded) { + elseif ($green -ge 1 -and -not $legAlsoFails) { + # Green on at least one sampled base build and red on none (failedCount == 0), and no + # earlier occurrence saw it red on base. The `-not $legAlsoFails` guard is order-independent: once + # ANY occurrence observed the leg red on base, a later green occurrence can no longer + # downgrade legBaselineResult to succeeded-on-base or re-arm legRegressed. + # Candidate regression -- but + # only CONFIRM it with enough base samples. A single green base build cannot + # distinguish a real regression from a UI test that merely happened to pass its one + # sampled base run; requiring several green base builds (MinBaseGreenSamples) removes + # that false positive. Deterministic build-error legs (crossgen/NativeAOT/linker/ + # MSBuild) compile or they don't, so one green base build is proof enough for those. $legBaselineResult = 'succeeded-on-base' - # Device-test TEST results suffer the XHarness exit-0 blind spot, so a test - # regression vs base is not trustworthy. A device-test BUILD break - # (crossgen/NativeAOT/linker/MSBuild) is deterministic -- the leg either compiled - # or it didn't -- so it IS a real regression even on a device-test pipeline. $legSource = [string](Get-ObjectValue -Object $occ -Names @("source")) - if (((-not $baseInfo.isDeviceTests) -or ($legSource -eq 'azdo-build-error')) -and (-not $isInfraProvisioning)) { + $eligible = (((-not $baseInfo.isDeviceTests) -or ($legSource -eq 'azdo-build-error')) -and (-not $isInfraProvisioning)) + $legIsDeterministicBuild = [bool](Get-ObjectValue -Object $occ -Names @("deterministicBuildError")) + # Only a DETERMINISTIC compile/toolchain break earns the single-green-base shortcut. A + # crash/OOM also carries source 'azdo-build-error' but is flaky, so it stays subject to + # MinBaseGreenSamples -- one lucky green base sample must not flip a flaky device-test crash + # to 'regressed-vs-base'. + $requiredGreen = if ($legSource -eq 'azdo-build-error' -and $legIsDeterministicBuild) { 1 } else { $MinBaseGreenSamples } + if ($eligible -and $green -ge $requiredGreen) { $legRegressed = $true } + elseif ($eligible) { + # Green on base but too few green samples to rule out flakiness -> NOT a + # confident regression. Leave legRegressed false so it flows to 'indeterminate' + # (NHI), and record why so the report can say "green on base, only N sample(s)". + $legBaselineResult = 'succeeded-on-base-unconfirmed' + } } } - # Cross-leg conflict veto: a failure that regressed cleanly in ONE leg (green on base) but - # was flaky on base in ANOTHER leg (inconclusive-on-base: failed an attempt, passed on - # retry) is NOT a trustworthy deterministic regression. The same failure text landing on a - # leg that is demonstrably flaky on base means the PR-red occurrence is most likely that - # same nondeterministic flake sprayed onto a different leg -- not a PR break. Suppress the + # Cross-leg conflict veto: a failure that regressed cleanly in ONE leg (green across base) + # but was ALSO red on base in ANOTHER leg (or in an earlier occurrence of the same leg) is + # NOT a trustworthy deterministic regression. Firing on $legAlsoFails (not just the flaky + # $legInconclusive) also closes the iteration-order hole where a later green occurrence set + # legRegressed=true after an earlier occurrence already saw the leg red on base. The same + # failure text landing on a leg that is red on base means the PR-red occurrence is most likely + # that same nondeterministic flake sprayed onto a different leg -- not a PR break. Suppress the # clean-regression claim and defer to a human (-> indeterminate / NHI). This never yields a # false green (the failure still forbids 'Ready to merge' via the indeterminate path); it # only stops over-claiming "regressed vs base" on flaky/environmental failures (e.g. an # Android 'platform-tools' provisioning flake that sprays across several legs at once). - if ($legInconclusive -and $legRegressed) { + # ($legInconclusive implies $legAlsoFails, so this subsumes the old flaky-only veto.) + if ($legAlsoFails -and $legRegressed) { $legRegressed = $false - $legBaselineResult = 'inconclusive-on-base' + $legBaselineResult = if ($legInconclusive) { 'flaky-on-base' } else { 'failed-on-base' } } $failure['legBaselineResult'] = $legBaselineResult $failure['legRegressedVsBase'] = [bool]$legRegressed $failure['legAlsoFailsOnBase'] = [bool]$legAlsoFails + # Base-sampling evidence for the report. baseSampleCount = how many recent base builds were read. + # baseGreenCount / baseFailedCount are the PER-LEG MAXIMA across this failure's occurrences (so on + # a multi-leg failure they may come from different legs and need not sum to baseSampleCount); the + # report labels them "per-leg max" for that reason. A confident 'regressed-vs-base' comes from a + # single clean leg, where the maxima equal that leg's counts: baseGreenCount >= MinBaseGreenSamples + # and baseFailedCount == 0 (any red on base trips the $legAlsoFails veto above). + $failure['baseSampleCount'] = [int]$baseSampleCount + $failure['baseGreenCount'] = [int]$baseGreenCount + $failure['baseFailedCount'] = [int]$baseFailedCount # Deterministic attribution prior the classifier MUST start from. Conservative precedence built to # never DISMISS a real PR break: only an EXACT test+platform match on base ('alsoFailsOnBaseline') @@ -2974,12 +3139,13 @@ foreach ($failure in $dedupedFailures) { if ($failure['matchesCiScan']) { # ...UNLESS the ci-scan registry documents this exact test (or its failing leg) as a # recurring/regressed/unstable failure on this base branch across MANY builds. The leg - # diff only saw the ONE most recent base build, which happened to be green; ci-scan's - # multi-build history shows the failure occurs on the base branch independent of this PR. - # DEMOTE the single-base regression claim to 'indeterminate' (NHI). This is a false-RED - # reduction ONLY: the failure still forbids a green verdict (it caps the ceiling at NHI), - # so a ci-scan hit -- an LLM-generated, possibly-stale hint -- can NEVER turn a red check - # green here; it can only move an over-confident 'Not ready' down to 'needs a human'. + # diff sampled only the last few base builds (on which the leg was green); ci-scan's + # deeper multi-build history shows the failure occurs on the base branch independent of + # this PR. DEMOTE the few-build regression claim to 'indeterminate' (NHI). This is a + # false-RED reduction ONLY: the failure still forbids a green verdict (it caps the ceiling + # at NHI), so a ci-scan hit -- an LLM-generated, possibly-stale hint -- can NEVER turn a + # red check green here; it can only move an over-confident 'Not ready' down to 'needs a + # human'. $failure['deterministicAttribution'] = 'indeterminate' $failure['ciScanDemoted'] = $true } @@ -3181,12 +3347,14 @@ $legsRegressedList = @($dedupedFailures | Where-Object { [string]$_.deterministi $legsRegressedVsBase = $legsRegressedList.Count $legsRegressedVsBaseNames = @($legsRegressedList | ForEach-Object { [string]$_.testName } | Select-Object -Unique) # Failures the deterministic prior could attribute NEITHER way: not a clean regression vs -# base, not pre-existing on base, not a known issue ('indeterminate'). Causes: the base leg -# outcome was ambiguous (a duplicate/retried leaf name -> 'inconclusive-on-base'), the base -# build was missing or unreadable, or a device-test TEST result outside the deterministic -# build-error class. We cannot prove these are PR-caused, but we equally cannot dismiss them -# as pre-existing/known -- so a green verdict is forbidden and they cap the ceiling at -# 'Needs human investigation' (softer than a proven regression's 'Not ready'). +# base, not pre-existing on base, not a known issue ('indeterminate'). Causes: the leg was +# flaky on base (red on some sampled base builds, green on others -> 'flaky-on-base'), the +# leg was green on base but on too few samples to confirm a regression +# ('succeeded-on-base-unconfirmed'), the base build was missing or unreadable, or a +# device-test TEST result outside the deterministic build-error class. We cannot prove these +# are PR-caused, but we equally cannot dismiss them as pre-existing/known -- so a green +# verdict is forbidden and they cap the ceiling at 'Needs human investigation' (softer than a +# proven regression's 'Not ready'). $unattributedList = @($dedupedFailures | Where-Object { [string]$_.deterministicAttribution -eq 'indeterminate' }) $unattributedFailures = $unattributedList.Count $unattributedFailureNames = @($unattributedList | ForEach-Object { [string]$_.testName } | Select-Object -Unique) @@ -3255,7 +3423,7 @@ elseif ($pendingChecks.Count -gt 0 -or $unmappedFailingChecks.Count -gt 0 -or $u $ceilingReasons.Add("$($unaccountedFailingChecks.Count) failing check(s) are backed by an accessible build that produced NO extractable failure and NO unexplained-leg record (a build/infra break whose log was unreadable, had no log id, or fell past the per-build cap); a 'Ready to merge' verdict is forbidden until a human reads them: $((@($unaccountedFailingChecks) | Select-Object -First 8) -join ', ').") } if ($unattributedFailures -gt 0) { - $ceilingReasons.Add("$unattributedFailures failure(s) could not be attributed deterministically (base outcome ambiguous, base build missing/unreadable, or a device-test result outside the build-error class); they are neither provably PR-caused nor dismissible as pre-existing/known, so a 'Ready to merge' verdict is forbidden until a human classifies them: $((@($unattributedFailureNames) | Select-Object -First 8) -join ', ').") + $ceilingReasons.Add("$unattributedFailures failure(s) could not be attributed deterministically (flaky on base, green on too few base samples to confirm a regression, base build missing/unreadable, or a device-test result outside the build-error class); they are neither provably PR-caused nor dismissible as pre-existing/known, so a 'Ready to merge' verdict is forbidden until a human classifies them: $((@($unattributedFailureNames) | Select-Object -First 8) -join ', ').") } if ($abortedFailingChecks.Count -gt 0) { $ceilingReasons.Add("$($abortedFailingChecks.Count) failing check(s) did not finish cleanly (cancelled/timed-out/startup-failure/stale/action-required); the result is not a trustworthy pass and the aborted legs may carry no extractable failure, so a 'Ready to merge' verdict is forbidden until a human reads them: $((@($abortedFailingChecks) | Select-Object -First 8) -join ', ').") @@ -3283,7 +3451,7 @@ else { # spot), so this fires only on trustworthy maui-pr build results -- exactly the crossgen/R2R class. if ($legsRegressedVsBase -gt 0 -and $verdictCeiling -in @('No failures found', 'Ready to merge', 'Needs human investigation')) { $verdictCeiling = "Not ready" - $ceilingReasons.Add("$legsRegressedVsBase leg/failure(s) are red on the PR but GREEN on the most recent completed base build (deterministic regression vs base): $((@($legsRegressedVsBaseNames) | Select-Object -First 8) -join ', '). A 'Ready to merge'/'No failures found' verdict is forbidden; the PR is at best 'Not ready'.") + $ceilingReasons.Add("$legsRegressedVsBase leg/failure(s) are red on the PR but GREEN on the sampled base builds and red on none (deterministic regression vs base): $((@($legsRegressedVsBaseNames) | Select-Object -First 8) -join ', '). A 'Ready to merge'/'No failures found' verdict is forbidden; the PR is at best 'Not ready'.") } if ($baselineInconclusiveRows -gt 0 -and $verdictCeiling -eq "Ready to merge") { $ceilingReasons.Add("$baselineInconclusiveRows baseline row(s) are inconclusive; do not subtract unmatched failures as pre-existing on baseline grounds alone.") @@ -3347,7 +3515,7 @@ if ($knownIssues.error) { $limitations.Add($knownIssues.error + " Known-issue cross-referencing was skipped; do not treat the absence of a known-issue match as evidence a failure is PR-caused.") } if ($ciScanIssues.error) { - $limitations.Add($ciScanIssues.error + " ci-scan multi-build base-branch cross-referencing was skipped; a single-base 'regressed-vs-base' could not be demoted by branch history, so treat such regressions as possibly-flaky pending a human check.") + $limitations.Add($ciScanIssues.error + " ci-scan multi-build base-branch cross-referencing was skipped; a few-build 'regressed-vs-base' could not be demoted by deeper branch history, so treat such regressions as possibly-flaky pending a human check.") } $context = [ordered]@{ @@ -3550,7 +3718,10 @@ else { $tag = if ($failure.ciScanDemoted) { " ⤵︎demoted" } else { "" } "[#$($failure.matchesCiScan.number)]($($failure.matchesCiScan.url)) ($($failure.matchesCiScan.class)/$($failure.matchesCiScan.matchKind))$tag" } else { "no" } - $legCell = if ($failure.legRegressedVsBase) { "REGRESSED" } elseif ($failure.legAlsoFailsOnBase) { "also-red" } elseif ($failure.legBaselineResult) { [string]$failure.legBaselineResult } else { "-" } + $legLabel = if ($failure.legRegressedVsBase) { "REGRESSED" } elseif ([string]$failure.legBaselineResult -eq 'flaky-on-base') { "flaky-on-base" } elseif ($failure.legAlsoFailsOnBase) { "also-red" } elseif ($failure.legBaselineResult) { [string]$failure.legBaselineResult } else { "-" } + $legCell = if ([int]$failure.baseSampleCount -gt 0) { + "$legLabel (per-leg max green $([int]$failure.baseGreenCount), max red $([int]$failure.baseFailedCount) across $([int]$failure.baseSampleCount) sampled base builds)" + } else { $legLabel } $attrCell = if ($failure.deterministicAttribution) { [string]$failure.deterministicAttribution } else { "indeterminate" } $md.Add("| $($failure.testName) | $($failure.platform) | $($failure.occurrenceCount) | $baseFlag | $legCell | $attrCell | $retryFlag | $knownIssueCell | $ciScanCell | $messages |") } diff --git a/.github/workflows/copilot-review-tests.lock.yml b/.github/workflows/copilot-review-tests.lock.yml index 00317f436f5c..9f4511ca6831 100644 --- a/.github/workflows/copilot-review-tests.lock.yml +++ b/.github/workflows/copilot-review-tests.lock.yml @@ -1,4 +1,4 @@ -# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"4ab52d614b79796382b8ac25207ef355ed42fb5ddcd4b37a53f87f2360677b0d","body_hash":"501c316db7803b2e8550b528e9dcfcca5a9270d1d971beccedf8bc2be6af5c39","compiler_version":"v0.80.9","strict":true,"agent_id":"copilot","agent_model":"claude-sonnet-4.6","engine_versions":{"copilot":"1.0.63"}} +# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"a660514f7d51959086570bf40aba54d37b75893084fab06731323da0498e3c96","body_hash":"3c792af463587f1966232513b60a511f14af7dc37f72f44645351cc0479eac76","compiler_version":"v0.80.9","strict":true,"agent_id":"copilot","agent_model":"claude-opus-4.8","engine_versions":{"copilot":"1.0.63"}} # gh-aw-manifest: {"version":1,"secrets":["COPILOT_PAT_0","COPILOT_PAT_1","COPILOT_PAT_2","COPILOT_PAT_3","COPILOT_PAT_4","COPILOT_PAT_5","COPILOT_PAT_6","COPILOT_PAT_7","COPILOT_PAT_8","COPILOT_PAT_9","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/cache/restore","sha":"27d5ce7f107fe9357f9df03efb73ab90386fccae","version":"v5.0.5"},{"repo":"actions/cache/save","sha":"27d5ce7f107fe9357f9df03efb73ab90386fccae","version":"v5.0.5"},{"repo":"actions/checkout","sha":"9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0","version":"v7.0.0"},{"repo":"actions/checkout","sha":"de0fac2e4500dabe0009e67214ff5f5447ce83dd","version":"v6.0.2"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-node","sha":"48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e","version":"v6.4.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"8c7d04ebf1ece56cd381446125da3e0f6896294a","version":"v0.80.9"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.27.7","digest":"sha256:aae231e4635c8999d039c132f1602d3df850fe9b84a00aa2b5ac981179b5661c","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.27.7@sha256:aae231e4635c8999d039c132f1602d3df850fe9b84a00aa2b5ac981179b5661c"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.7","digest":"sha256:009caf2e3d88fa77b64e9a03a95a228fc58db0f1701c6d324b29ba5a3c7c79b6","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.7@sha256:009caf2e3d88fa77b64e9a03a95a228fc58db0f1701c6d324b29ba5a3c7c79b6"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.27.7","digest":"sha256:deb1d4e19de62d51cee0508057a596a19315c3423ada4d675cad136dc8037c96","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.27.7@sha256:deb1d4e19de62d51cee0508057a596a19315c3423ada4d675cad136dc8037c96"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.3.27","digest":"sha256:fe984bddde4ec05d756d9043edb0a32912e6b7b72f6a121b1082f29221421cc7","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.3.27@sha256:fe984bddde4ec05d756d9043edb0a32912e6b7b72f6a121b1082f29221421cc7"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:529d02eb970b1161aa25c593a9c3df57fdfad5a8add328cb3b6eccef66f3183b","pinned_image":"ghcr.io/github/gh-aw-node@sha256:529d02eb970b1161aa25c593a9c3df57fdfad5a8add328cb3b6eccef66f3183b"},{"image":"ghcr.io/github/github-mcp-server:v1.4.0","digest":"sha256:2afb26356481d1a350e14544a6e160f7f7ec1561a1ea309b823665abf0309036","pinned_image":"ghcr.io/github/github-mcp-server:v1.4.0@sha256:2afb26356481d1a350e14544a6e160f7f7ec1561a1ea309b823665abf0309036"}]} # This file was automatically generated by gh-aw (v0.80.9). DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md # @@ -199,7 +199,7 @@ jobs: env: GH_AW_INFO_ENGINE_ID: "copilot" GH_AW_INFO_ENGINE_NAME: "GitHub Copilot CLI" - GH_AW_INFO_MODEL: "claude-sonnet-4.6" + GH_AW_INFO_MODEL: "claude-opus-4.8" GH_AW_INFO_VERSION: "1.0.63" GH_AW_INFO_AGENT_VERSION: "1.0.63" GH_AW_INFO_CLI_VERSION: "v0.80.9" @@ -956,7 +956,7 @@ jobs: needs.pat_pool.outputs.pat_number == '9', secrets.COPILOT_PAT_9, 'NO COPILOT PAT AVAILABLE') }} - COPILOT_MODEL: claude-sonnet-4.6 + COPILOT_MODEL: claude-opus-4.8 GH_AW_MAX_AI_CREDITS: ${{ vars.GH_AW_DEFAULT_MAX_AI_CREDITS || '1000' }} GH_AW_MAX_TURNS: ${{ vars.GH_AW_DEFAULT_MAX_TURNS || '' }} GH_AW_PHASE: agent @@ -1573,7 +1573,7 @@ jobs: needs.pat_pool.outputs.pat_number == '9', secrets.COPILOT_PAT_9, 'NO COPILOT PAT AVAILABLE') }} - COPILOT_MODEL: claude-sonnet-4.6 + COPILOT_MODEL: claude-opus-4.8 GH_AW_MAX_AI_CREDITS: ${{ vars.GH_AW_DEFAULT_DETECTION_MAX_AI_CREDITS || '400' }} GH_AW_MAX_TURNS: ${{ vars.GH_AW_DEFAULT_MAX_TURNS || '' }} GH_AW_PHASE: detection @@ -1863,7 +1863,7 @@ jobs: GH_AW_DETECTION_REASON: ${{ needs.detection.outputs.detection_reason }} GH_AW_EFFECTIVE_TOKENS: ${{ needs.agent.outputs.effective_tokens }} GH_AW_ENGINE_ID: "copilot" - GH_AW_ENGINE_MODEL: "claude-sonnet-4.6" + GH_AW_ENGINE_MODEL: "claude-opus-4.8" GH_AW_ENGINE_VERSION: "1.0.63" GH_AW_THREAT_DETECTION_AIC: ${{ needs.detection.outputs.aic }} GH_AW_WORKFLOW_ID: "copilot-review-tests" diff --git a/.github/workflows/copilot-review-tests.md b/.github/workflows/copilot-review-tests.md index d2d03acfa008..f8bfcf42f741 100644 --- a/.github/workflows/copilot-review-tests.md +++ b/.github/workflows/copilot-review-tests.md @@ -143,7 +143,7 @@ permissions: engine: id: copilot - model: claude-sonnet-4.6 + model: claude-opus-4.8 env: COPILOT_GITHUB_TOKEN: | ${{ case( @@ -243,9 +243,11 @@ steps: Invoke the **review-test-failures** skill: read and follow `.github/skills/review-test-failures/SKILL.md`. +**Comment-format precedence (gh-aw path):** for this workflow's posted PR comment, the format defined below — the badge row, the collapsible, and the succinct root-cause-grouped bullet list — **overrides** the skill's own output contract (the per-failure Markdown table, Platform column, nested evidence `
`, and glyph rules) wherever they differ. Use the skill only for its gathering steps, evidence fields, and verdict logic; render the result in the format specified here, never the skill's table. (The skill's original output contract still governs its local/standalone runner, which does not post through this workflow.) + That skill also references the canonical `.github/docs/maui-ci-facts.md`. The end goal is one **overall merge-readiness verdict** (Ready to merge / Not ready / Needs human investigation / Insufficient data / No failures found), informed by a **baseline comparison** against the most recent base-branch build. Use the gathered `failures.baseline`, `failures.baselineMatchCount`, `alsoFailsOnBaseline`, and `baselineSummary` fields — do not treat a failure as pre-existing without that evidence. -The gathered `context.json`/`context.md` also carry a deterministic **merge-readiness gate** (`gate.verdictCeiling`, `gate.ceilingReasons`, coverage counts) plus per-failure `matchesKnownIssue`, `retriedStillFailing`, and the computed **job-level baseline diff** (`legBaselineResult` / `legRegressedVsBase` / `legAlsoFailsOnBase` and a `deterministicAttribution` prior) evidence. Build-job breaks with no test name (crossgen/ReadyToRun, NativeAOT/ILC, linker, MSBuild `error`, and fatal non-coded breaks — native crash/segfault/OOM, test-host crash, unhandled exception) are extracted as distinct failures too (`source = azdo-build-error`), and any failed build leg that yields **no** extractable failure is counted in `gate.unexplainedFailedLegs`. A leg that is red on the PR but green on the same leg of the most recent base build is a computed regression in `gate.legsRegressedVsBase`. An accessible failing check that yields **no** extractable failure and **no** unexplained-leg record is counted in `gate.unaccountedFailingChecks` (the earned-green guard). A failing check whose GitHub conclusion did not finish cleanly (`CANCELLED`/`TIMED_OUT`/`STARTUP_FAILURE`/`STALE`/`ACTION_REQUIRED`) is counted in `gate.abortedFailingChecks` — its aborted legs can carry no `error` issue, so a PR-induced hang/cancellation must not be masked green by a dismissible sibling on the same build. A backing build whose **own result is `canceled`** (regardless of the GitHub check conclusion) is counted in `gate.canceledBuildChecks` — broader than the conclusion-based guard, it catches a build canceled mid-flight after a leg already posted `FAILURE`/`SUCCESS`. A **green device-test check** (`maui-pr-devicetests`) whose `Failed == 0` could not be positively confirmed is counted in `gate.deviceTestUnverified` — XHarness exits 0 even when device tests fail, so a green device-test check is trusted only when a fail count was observed all-zero over a **complete, error-free read** (Helix aggregated with every discovered job read without a thrown error, or the authenticated test-API paged through all runs and never trusting a truncated run set). A failure the prior can attribute neither way (base outcome ambiguous, base build missing/unreadable, or a device-test result outside the build-error class) is counted in `gate.unattributedFailures`; a `pre-existing-on-base`/`known-issue` dismissal is refused (downgraded to `indeterminate`) when the PR edits the failing test file (`scopeGuardTripped`) or when the PR and base failures of the same test have a reason conflict (`baselineReasonConflict` — reasons differ, with wrapper exceptions unwrapped to the inner cause (multiple inner exceptions collapsed to a sorted compound token), a normalized message fingerprint absent from base (the fingerprint keeps identifier-internal digits and hashes any long tail so distinct breaks stay distinct), or — for a test failure that exposes no reason and no message at all — zero corroboration that it is the same failure as the name match). Your overall verdict **MUST NOT be more favorable than `gate.verdictCeiling`** — a green verdict is impossible while a check is pending, a failing check could not be inspected, `gate.unexplainedFailedLegs > 0`, `gate.unaccountedFailingChecks > 0`, `gate.abortedFailingChecks > 0`, `gate.canceledBuildChecks > 0`, `gate.deviceTestUnverified > 0`, or `gate.unattributedFailures > 0`, and the ceiling is capped at `Not ready` whenever `gate.legsRegressedVsBase > 0`. Treat a `deterministicAttribution = regressed-vs-base` failure as Likely PR-caused unless you can cite why the base comparison is invalid. Only dismiss a failure as pre-existing when `deterministicAttribution` is `pre-existing-on-base` (exact test+platform also red on base) or `known-issue` (the **exact same test+platform also failed on base** AND the message matches a known issue — a richer label for the same dismissable case; leg-level corroboration is too coarse and no longer dismisses); a leg-only base match (`legAlsoFailsOnBase` with `deterministicAttribution = indeterminate`), an **uncorroborated** `matchesKnownIssue` hit (no exact base match), a `baselineReasonConflict` failure, or a `succeeded-on-base` device-test leg whose regression was suppressed is NOT dismissable and is already counted in `gate.unattributedFailures`. Build-job breaks are extracted even on a leg that also has a test failure (a pre-existing flaky test cannot hide a new build break), `partiallySucceeded` records are inspected on both sides like `failed`, and a baseline dismissal is **scoped to the same pipeline definition** (a failure in one pipeline is never dismissed by a same-named base failure from another). Surface the coverage ledger and ceiling in the report so the verdict is provably sound. +The gathered `context.json`/`context.md` also carry a deterministic **merge-readiness gate** (`gate.verdictCeiling`, `gate.ceilingReasons`, coverage counts) plus per-failure `matchesKnownIssue`, `retriedStillFailing`, and the computed **job-level baseline diff** (`legBaselineResult` / `legRegressedVsBase` / `legAlsoFailsOnBase` and a `deterministicAttribution` prior) evidence. Build-job breaks with no test name (crossgen/ReadyToRun, NativeAOT/ILC, linker, MSBuild `error`, and fatal non-coded breaks — native crash/segfault/OOM, test-host crash, unhandled exception) are extracted as distinct failures too (`source = azdo-build-error`), and any failed build leg that yields **no** extractable failure is counted in `gate.unexplainedFailedLegs`. A leg that is red on the PR but green across several recent base builds (of the PR's own base branch — `main` or `net11.0`) and red on none of them is a computed regression in `gate.legsRegressedVsBase` (a **deterministic** build break — crossgen/NativeAOT/linker/MSBuild — needs only one green base build, since it compiles or it doesn't). An accessible failing check that yields **no** extractable failure and **no** unexplained-leg record is counted in `gate.unaccountedFailingChecks` (the earned-green guard). A failing check whose GitHub conclusion did not finish cleanly (`CANCELLED`/`TIMED_OUT`/`STARTUP_FAILURE`/`STALE`/`ACTION_REQUIRED`) is counted in `gate.abortedFailingChecks` — its aborted legs can carry no `error` issue, so a PR-induced hang/cancellation must not be masked green by a dismissible sibling on the same build. A backing build whose **own result is `canceled`** (regardless of the GitHub check conclusion) is counted in `gate.canceledBuildChecks` — broader than the conclusion-based guard, it catches a build canceled mid-flight after a leg already posted `FAILURE`/`SUCCESS`. A **green device-test check** (`maui-pr-devicetests`) whose `Failed == 0` could not be positively confirmed is counted in `gate.deviceTestUnverified` — XHarness exits 0 even when device tests fail, so a green device-test check is trusted only when a fail count was observed all-zero over a **complete, error-free read** (Helix aggregated with every discovered job read without a thrown error, or the authenticated test-API paged through all runs and never trusting a truncated run set). A failure the prior can attribute neither way (flaky on base, green on too few base samples to confirm a regression — `succeeded-on-base-unconfirmed`, base build missing/unreadable, or a device-test result outside the build-error class) is counted in `gate.unattributedFailures`; a `pre-existing-on-base`/`known-issue` dismissal is refused (downgraded to `indeterminate`) when the PR edits the failing test file (`scopeGuardTripped`) or when the PR and base failures of the same test have a reason conflict (`baselineReasonConflict` — reasons differ, with wrapper exceptions unwrapped to the inner cause (multiple inner exceptions collapsed to a sorted compound token), a normalized message fingerprint absent from base (the fingerprint keeps identifier-internal digits and hashes any long tail so distinct breaks stay distinct), or — for a test failure that exposes no reason and no message at all — zero corroboration that it is the same failure as the name match). Your overall verdict **MUST NOT be more favorable than `gate.verdictCeiling`** — a green verdict is impossible while a check is pending, a failing check could not be inspected, `gate.unexplainedFailedLegs > 0`, `gate.unaccountedFailingChecks > 0`, `gate.abortedFailingChecks > 0`, `gate.canceledBuildChecks > 0`, `gate.deviceTestUnverified > 0`, or `gate.unattributedFailures > 0`, and the ceiling is capped at `Not ready` whenever `gate.legsRegressedVsBase > 0`. Treat a `deterministicAttribution = regressed-vs-base` failure as Likely PR-caused unless you can cite why the base comparison is invalid. Only dismiss a failure as pre-existing when `deterministicAttribution` is `pre-existing-on-base` (exact test+platform also red on base) or `known-issue` (the **exact same test+platform also failed on base** AND the message matches a known issue — a richer label for the same dismissable case; leg-level corroboration is too coarse and no longer dismisses); a leg-only base match (`legAlsoFailsOnBase` with `deterministicAttribution = indeterminate`), an **uncorroborated** `matchesKnownIssue` hit (no exact base match), a `baselineReasonConflict` failure, or a `succeeded-on-base` device-test leg whose regression was suppressed is NOT dismissable and is already counted in `gate.unattributedFailures`. Build-job breaks are extracted even on a leg that also has a test failure (a pre-existing flaky test cannot hide a new build break), `partiallySucceeded` records are inspected on both sides like `failed`, and a baseline dismissal is **scoped to the same pipeline definition** (a failure in one pipeline is never dismissed by a same-named base failure from another). Surface the coverage ledger and ceiling in the report so the verdict is provably sound. ## Target @@ -287,13 +289,7 @@ When triggered via `workflow_dispatch`, `${{ inputs.suppress_output }}` controls ## When no failures are found -If the gathered context shows no failing, pending, or inconclusive checks and no extracted failures, still post a PR conversation comment with `add_comment` unless dry-run mode is active. Use the same collapsed shape as other results with: - -- Overall verdict: `No failures found` -- Overall badge color: `1a7f37` -- Failures badge value: `0` -- No platform badges -- Recommended action: no test-failure action is needed +If the gathered context shows no failing, pending, or inconclusive checks and no extracted failures, still post a PR conversation comment with `add_comment` unless dry-run mode is active. Use the same shape as other results — the badge row (`Overall` = `No failures found` in `1a7f37`, `Failures` = `0`, `Regressed vs base` = `0`, `Baseline` = `0 on base`) and the collapsible with an `**Overall verdict:** No failures found …` line, no grouped bullets, and a `Recommended action` of no test-failure action is needed. Only call `noop` when dry-run mode is active and no PR comment should be posted. @@ -310,46 +306,41 @@ If dry-run mode is not active, call `add_comment` exactly once with `item_number > To request a fresh review after new comments, commits, or CI runs, comment `/review tests`.

- Overall [verdict] + Overall [verdict] Failures [count] - Baseline [n on base] - Platform [platform] + Regressed vs base [n] + Baseline [m on base]

Test Failure Review: [verdict] - click to expand -**Overall verdict:** [Ready to merge | Not ready | Needs human investigation | Insufficient data | No failures found] +**Overall verdict:** [one or two sentences summarizing the strongest evidence — how many distinct failures are genuine regressions vs the base branch, and how many are pre-existing or flaky-on-base. Name the base branch (`main` / `net11.0`) and how many recent base builds were actually sampled — use the real `baseSampleCount` (say "N recent base builds", or "the single readable base build" when only one); do not hard-code "several".] -[One or two sentences summarizing the strongest evidence, including how many failures are pre-existing on the base branch.] +- **✗ PR-related** — [root-cause group label] (~[N] tests): [one sentence tying the group to the PR's changed area or a shared failure pattern; name at most ONE representative test in `code`]. +- **ℹ Uncertain** — [root-cause group label] (~[N] legs): [one sentence — unexplained build legs, aborted/canceled checks, unattributed, leg-only flaky-on-base, or device-test-unverified]. +- **● Unrelated** — [root-cause group label] (~[N] tests): [one sentence — pre-existing-on-base / known issue]. **Coverage:** [gate.totalChecks] checks · [passingOrNeutralChecks] passing · [failingChecks] failing · [pendingChecks] pending · [inaccessibleFailingChecks] inaccessible · [unmappedFailingChecks] unmapped · [unexplainedFailedLegs] unexplained build legs · [unaccountedFailingChecks] unaccounted failing checks · [abortedFailingChecks] aborted failing checks · [canceledBuildChecks] canceled-build checks · [deviceTestUnverified] device-test unverified · [unattributedFailures] unattributed · [legsRegressedVsBase] regressed-vs-base. Deterministic ceiling: [gate.verdictCeiling][ — reason from gate.ceilingReasons when present]. -| Failure | Verdict | On base? | Evidence | -| --- | --- | --- | --- | -| [check/test/build] | [Likely PR-caused | Likely unrelated | Needs human investigation | Insufficient data] | [yes/no — "regressed" when legRegressedVsBase, "also-red" when legAlsoFailsOnBase, else alsoFailsOnBaseline] | [specific evidence — lead with deterministicAttribution when regressed-vs-base/pre-existing-on-base, cite a known-issue link when matchesKnownIssue is set, note "retried still failing" when true, link build/test IDs] | +**Builds (this PR):** [build definition + ID links]. **Base sampling ([base branch], [N] recent build(s) per definition — the actual `baseSampleCount`):** [recent base build ID links]. ### Recommended action [One concise recommendation.] -
-Evidence details - -[Relevant checks, build IDs, baseline build IDs, test run IDs, log excerpts, PR-scope details, and limitations (including when baseline data was unavailable).] - -
-
``` -The `Overall` badge and `**Overall verdict:**` line carry the merge-readiness verdict. The per-failure table carries the per-failure verdicts plus an `On base?` column (driven by the computed job-level diff: "regressed" when `legRegressedVsBase`, "also-red" when `legAlsoFailsOnBase`, otherwise yes/no from `alsoFailsOnBaseline`). The `**Coverage:**` line reports the deterministic gate counts and `gate.verdictCeiling`; the overall verdict must never be more favorable than that ceiling. Overall badge colors: `1a7f37` for `Ready to merge` and `No failures found`, `d1242f` for `Not ready`, `bf8700` for `Needs human investigation`, `6e7781` for `Insufficient data`. +The comment stays compact when collapsed: the badge row plus the `
` summary line carry the at-a-glance verdict, and all detail lives inside the collapsible. Badges (in order): `Overall` = the merge-readiness verdict; `Failures` = distinct-failure count; `Regressed vs base` = `gate.legsRegressedVsBase` (the PR-caused count); `Baseline` = how many distinct failures also appear on the base branch (dismissable). Badge colors: `Overall` uses `1a7f37` for `Ready to merge`/`No failures found`, `d1242f` for `Not ready`, `bf8700` for `Needs human investigation`, `6e7781` for `Insufficient data`; `Regressed vs base` is `d1242f` when > 0 and `1a7f37` when 0; `Failures` is `8250df`; `Baseline` is `0969da`. -Do not apply labels, trigger reruns, approve the PR, request changes, or modify code. +Inside the collapsible, replace the old per-test table with a **succinct root-cause-grouped bullet list** (the deep UI-failure-analysis shape) — **each bullet is one root-cause GROUP, never one bullet per test** (a run can have hundreds of failures — long name lists are unreadable). Do **not** emit a Markdown table and do **not** list every failing test name. Begin every bullet with exactly one assessment token in bold — **✗ PR-related** (a `deterministicAttribution = regressed-vs-base` failure: red on the PR but green across several recent base builds and red on none — treat as Likely PR-caused unless you can cite why the base comparison is invalid), **ℹ Uncertain** (`indeterminate` / an unexplained or aborted/canceled leg / `unattributedFailures` / a leg-only `flaky-on-base` failure whose `deterministicAttribution` is `indeterminate` / device-test-unverified), or **● Unrelated** (only a `pre-existing-on-base` or `known-issue` failure — `deterministicAttribution` takes precedence over `legBaselineResult`, so a leg-only `flaky-on-base` with `deterministicAttribution = indeterminate` is **ℹ Uncertain**, not Unrelated). Then " — " a short human label for the group plus an approximate count in parentheses (e.g. "(~20 tests)"), then ": " a one-sentence why; name at most ONE representative test in `code`. Order the bullets ✗ PR-related first, then ℹ Uncertain, then ● Unrelated; aim for at most ~6 bullets and merge groups that share a root cause. -Do not include a Data badge. +The `**Overall verdict:**` line and the `Overall` badge carry the merge-readiness verdict and **must never be more favorable than `gate.verdictCeiling`**. The `**Coverage:**` line reports the deterministic gate counts and `gate.verdictCeiling`. Allowed verdicts: `Ready to merge`, `Not ready`, `Needs human investigation`, `Insufficient data`, `No failures found`. + +Do not apply labels, trigger reruns, approve the PR, request changes, or modify code. -Do not use emojis anywhere in the posted comment. +Do not use colorful emojis anywhere in the posted comment; the only status glyphs are the subtle tokens ✗, ●, and ℹ. Use Markdown links, not raw `` tags. gh-aw safe outputs sanitize raw anchors before posting.