Skip to content
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions .github/docs/maui-ci-facts.md
Original file line number Diff line number Diff line change
Expand Up @@ -271,10 +271,13 @@ modifies that visual test or the affected snapshot/platform.
The automated `/review tests` lane gathers failed UI result IDs from the public
`vstmr.dev.azure.com/.../testresults/resultsbybuild` endpoint, then reads the public
result-detail and attachment APIs. It publishes validated baseline/actual/diff PNGs to
the repository's `review-tests-assets` branch. A trusted post-step inserts as many
complete expandable comparison panels as fit inside the single test-failure analysis
comment while enforcing gh-aw's URL, mention, and character limits; excess panels are
reported as omitted rather than creating another comment. Visual publishing is
the repository's orphan, asset-only `review-tests-assets-v2` branch. The legacy
`review-tests-assets` branch remains intact so existing commit-pinned image URLs stay
reachable; it is not used for new publications because its inherited repository tree
contains workflow files that `GITHUB_TOKEN` cannot update. A trusted post-step inserts
as many complete expandable comparison panels as fit inside the single test-failure
analysis comment while enforcing gh-aw's URL, mention, and character limits; excess
panels are reported as omitted rather than creating another comment. Visual publishing is
supplementary evidence only: missing images never raise or lower the deterministic
verdict ceiling. Each panel also shows a conservative relationship label derived from
the exact test-and-platform `deterministicAttribution` plus exact changed snapshot/test
Expand Down
361 changes: 361 additions & 0 deletions .github/scripts/Review-Tests.Tests.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ BeforeAll {
foreach ($functionName in @(
'Invoke-SealedVisualMerge',
'Get-EmbeddedTestFailureReport',
'Get-EmbeddedTestFailureReportCandidate',
'Get-MarkdownFenceState',
'Escape-Html',
'Get-ReportVerdict',
Expand Down Expand Up @@ -126,6 +127,366 @@ Trailing assistant prose.
Should -Be ([regex]::Matches($report, '</details>').Count)
}

It 'ignores a marker quoted in assistant prose before the standalone report marker' {
$content = @'
The report could not be written. Per the fallback rule ("return the report beginning with
`<!-- Tests Failure -->`"), here is the complete report:

<!-- Tests Failure -->

## Tests Failure Analysis

<details>
<summary>Review</summary>
Evidence
</details>
'@

$body = New-TestFailureReviewBody `
-PRNumber 1 `
-Repository 'dotnet/maui' `
-ReportContent $content `
-ContextJsonPath 'unused.json'

$body | Should -Not -Match 'fallback rule|here is the complete report'
$body | Should -Match '^<!-- Tests Failure \(local\) -->'
([regex]::Matches($body, '<!-- Tests Failure \(local\) -->').Count) | Should -Be 1
}

It 'continues past a standalone marker in an earlier fenced example' {
$content = @'
The required output shape is:

```markdown
<!-- Tests Failure -->
Example only.
```

The actual report follows.

<!-- Tests Failure -->

## Tests Failure Analysis

<details>
<summary>Actual review</summary>
Actual evidence
</details>
'@

$report = Get-EmbeddedTestFailureReport -Content $content

$report | Should -Match '^<!-- Tests Failure -->'
$report | Should -Match 'Actual evidence'
$report | Should -Not -Match 'Example only|actual report follows'
}

It 'does not let an earlier unfenced example marker borrow the later report structure' {
$content = @'
<!-- Tests Failure -->
Example only.

<!-- Tests Failure -->

## Tests Failure Analysis

<details>
<summary>Actual review</summary>
Actual evidence
</details>
'@

$body = New-TestFailureReviewBody `
-PRNumber 1 `
-Repository 'dotnet/maui' `
-ReportContent $content `
-ContextJsonPath 'unused.json'

$body | Should -Not -Match 'Example only'
$body | Should -Match 'Actual evidence'
([regex]::Matches($body, '<!-- Tests Failure \(local\) -->').Count) | Should -Be 1
}

It 'prefers a standalone marker over an earlier explanatory report heading' {
$content = @'
## Tests Failure Analysis

This heading only explains the output that follows.

<!-- Tests Failure -->

## Tests Failure Analysis

<details>
<summary>Actual review</summary>
Actual evidence
</details>
'@

$report = Get-EmbeddedTestFailureReport -Content $content

$report | Should -Match '^<!-- Tests Failure -->'
$report | Should -Match 'Actual evidence'
$report | Should -Not -Match 'only explains'
}
Comment thread
kubaflo marked this conversation as resolved.
Resolved

It 'prefers the earliest real report over a later structurally-valid fenced duplicate' {
Comment thread
kubaflo marked this conversation as resolved.
Resolved
# Regression guard for the reverse-iteration tie-break: a real report FOLLOWED by a
# complete, fenced copy of the template must still return the first (real) block. The
# extracted content is published and can quote untrusted material, so a trailing
# well-formed duplicate must never displace the real verdict.
$content = @'
<!-- Tests Failure -->

## Tests Failure Analysis

<details>
<summary>Actual review</summary>
Actual evidence
</details>

Then the assistant echoes the expected shape as a fenced example:

```markdown
<!-- Tests Failure -->

## Tests Failure Analysis

<details>
<summary>Template example</summary>
Example evidence
</details>
```
'@

$report = Get-EmbeddedTestFailureReport -Content $content

$report | Should -Match '^<!-- Tests Failure -->'
$report | Should -Match 'Actual evidence'
$report | Should -Not -Match 'Template example|Example evidence'
}

It 'keeps a real report that quotes the marker inside its own details block (tier 1)' {
# Round-3 regression guard (❌ 3/3). A real report that quotes the required template
# marker inside its own fenced code — before its closing </details> — must NOT be
# truncated. Bounding at the raw next anchor sliced off the report's own </details>,
# the details-balance never returned to 0, and the candidate collapsed to $null —
# which New-TestFailureReviewBody silently replaces with a synthesized skeleton,
# the exact content-loss this PR exists to prevent. The quoted marker sits inside
# code, so it is not a structural sibling and must never bound the report.
$content = @'
<!-- Tests Failure -->

## Tests Failure Analysis

<details>
<summary>Actual review</summary>
Actual evidence.

The expected template shape is:

```markdown
<!-- Tests Failure -->

## Tests Failure Analysis
```

More real evidence after the example.
</details>
'@

$report = Get-EmbeddedTestFailureReport -Content $content

$report | Should -Not -BeNullOrEmpty
$report | Should -Match '^<!-- Tests Failure -->'
$report | Should -Match 'Actual evidence'
$report | Should -Match 'More real evidence after the example'
}

It 'keeps a heading-anchored report that quotes the heading inside its own details block (tier 2)' {
# Tier-2 counterpart: with no HTML marker, `## Tests Failure Analysis` anchors the
# report. A heading quoted inside the report's own fenced code must not bound it
# either — the same structural-anchor rule applies to the heading tier.
$content = @'
## Tests Failure Analysis

<details>
<summary>Actual review</summary>
Actual evidence.

Reports must start with:

```markdown
## Tests Failure Analysis
```

Final evidence line.
</details>
'@

$report = Get-EmbeddedTestFailureReport -Content $content

$report | Should -Not -BeNullOrEmpty
$report | Should -Match '^## Tests Failure Analysis'
$report | Should -Match 'Actual evidence'
$report | Should -Match 'Final evidence line'
}

It 'keeps a real report that quotes the marker as a bare unfenced line in its own details (tier 1)' {
# Round-4 follow-up: the quoted marker is a STANDALONE UNFENCED line (0-3 space indent),
# not fenced or 4-space-indented, sitting inside the report's own still-open <details>.
# The structural (non-depth-aware) bound treated it as a sibling and collapsed the
# report to $null; the depth-aware bound recognizes depth > 0 means it's the report's
# own content and continues to the report's real closing </details>.
$content = @'
<!-- Tests Failure -->

## Tests Failure Analysis

<details>
<summary>Actual review</summary>
Actual evidence.

The report must begin with this exact line:

<!-- Tests Failure -->

and everything after it is analysis. More real evidence follows.
</details>
'@

$report = Get-EmbeddedTestFailureReport -Content $content

$report | Should -Not -BeNullOrEmpty
$report | Should -Match '^<!-- Tests Failure -->'
$report | Should -Match 'Actual evidence'
$report | Should -Match 'More real evidence follows'
}

It 'keeps a heading-anchored report that quotes the heading as a bare unfenced line in its own details (tier 2)' {
$content = @'
## Tests Failure Analysis

<details>
<summary>Actual review</summary>
Actual evidence.

Every report opens with the heading:

## Tests Failure Analysis

which is just quoted here. Final evidence line.
</details>
'@

$report = Get-EmbeddedTestFailureReport -Content $content

$report | Should -Not -BeNullOrEmpty
$report | Should -Match '^## Tests Failure Analysis'
$report | Should -Match 'Actual evidence'
$report | Should -Match 'Final evidence line'
}

It 'returns the earliest report when a self-quoting first report precedes a genuine second (no silent substitution)' {
# The sharpest round-4 case: a self-quoting report #1 (bare unfenced marker inside its
# own <details>) FOLLOWED by a genuine report #2. The structural bound truncated #1 to
# $null, then returned #2 — silently dropping #1 and violating earliest-wins. Depth-aware
# bounding keeps #1 intact and returns it.
$content = @'
<!-- Tests Failure -->

## Tests Failure Analysis

<details>
<summary>Report ONE</summary>
Evidence for report ONE.

Template reference line:

<!-- Tests Failure -->

end of the quoted template.
</details>

Then, later, a genuinely separate second report:

<!-- Tests Failure -->

## Tests Failure Analysis

<details>
<summary>Report TWO</summary>
Evidence for report TWO.
</details>
'@

$report = Get-EmbeddedTestFailureReport -Content $content

$report | Should -Match '^<!-- Tests Failure -->'
$report | Should -Match 'Evidence for report ONE'
$report | Should -Not -Match 'Evidence for report TWO'
}

It 'does not let an unclosed earlier report borrow a later report''s closing tag' {
Comment thread
kubaflo marked this conversation as resolved.
Resolved
# PureWeen's dispute-round caveat: with depth-aware bounding, an earlier report whose
# <details> is never closed must NOT borrow a later genuine report's </details> to
# complete itself. The balance loop rejects the unbalanced first candidate (depth never
# returns to 0), so extraction falls through to the genuine second report cleanly.
$content = @'
<!-- Tests Failure -->

## Tests Failure Analysis

<details>
<summary>Unclosed report one</summary>
Evidence one — this details block is never closed.

<!-- Tests Failure -->

## Tests Failure Analysis

<details>
<summary>Report two</summary>
Evidence two.
</details>
'@

$report = Get-EmbeddedTestFailureReport -Content $content

$report | Should -Match 'Evidence two'
$report | Should -Not -Match 'Evidence one'
}

It 'recognizes a standalone marker indented up to three spaces' {
# Column-0-only anchoring missed markers re-indented by list nesting / wrapping.
# 0-3 spaces is still a paragraph-level standalone line, so it must anchor.
$content = @'
<!-- Tests Failure -->

## Tests Failure Analysis

<details>
<summary>Indented marker</summary>
Indented evidence
</details>
'@

$report = Get-EmbeddedTestFailureReport -Content $content

$report | Should -Match '^<!-- Tests Failure -->'
$report | Should -Match 'Indented evidence'
}

It 'ignores a marker indented four or more spaces (indented code block)' {
# 4-space indent = CommonMark indented code block; must NOT anchor. Guards against
# loosening the anchor to an unbounded [ \t]* (which would match markers in code).
$content = "Example, shown as an indented code block:`n`n <!-- Tests Failure -->`n`n<details>`n<summary>Code sample</summary>`nSample evidence`n</details>"

$report = Get-EmbeddedTestFailureReport -Content $content

$report | Should -BeNullOrEmpty
}

It 'preserves code fences inside an unfenced report and trims trailing prose' {
$content = @'
The write was denied, so the report is below.
Expand Down
Loading
Loading