Skip to content
Merged
Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
e07e066
Automate visual failure comparisons in /review tests
Copilot Jul 19, 2026
b9f1f73
Merge visual comparisons into /review tests
Copilot Jul 19, 2026
87ce58d
Label visual failures by PR relationship
Copilot Jul 20, 2026
4ccbab1
Use changed visual scope in relation labels
Copilot Jul 20, 2026
8e606da
Fix report-fence parsing + visual dedup key; align review-tests auth …
kubaflo Jul 21, 2026
5cb72ca
Harden visual-asset publishing per review feedback
kubaflo Jul 21, 2026
96b9fe0
Unset GH_AW_* tokens before the trusted review-tests merge step
kubaflo Jul 21, 2026
294ed45
Fix visual-evidence dedup, isolate diff download, surface prep failures
kubaflo Jul 21, 2026
88eb005
Make /review tests pre-activation resilient so the failure-report fal…
kubaflo Jul 21, 2026
e48a947
Merge main into review-tests-visual-comparisons
Copilot Jul 22, 2026
abddbf5
Harden visual-evidence gather/publish/merge per review feedback
kubaflo Jul 22, 2026
f4cf6cf
Bound remaining AzDO/Helix Invoke-WebRequest calls with -TimeoutSec
Copilot Jul 22, 2026
42cd619
Harden /review tests visual budgeting, leg dedup, and omission accoun…
Copilot Jul 22, 2026
fe5e030
Harden visual budget lifetime, download cancellation, and omission ac…
Copilot Jul 22, 2026
18d5f31
Distinguish publish failures and omission causes; charge visual budge…
Copilot Jul 23, 2026
7f2018c
Harden visual-publish failure classification and per-request timeouts…
Copilot Jul 23, 2026
a438e1d
Bound visual result pagination by deadline
Copilot Jul 23, 2026
ae857e4
Harden visual attribution and workflow bounds
Copilot Jul 23, 2026
aac848e
Clean up rejected baseline candidates and cover near-expiry paginatio…
kubaflo Jul 23, 2026
3ab5098
Seal local visual merge and enforce bounds
Copilot Jul 23, 2026
7ec3c32
Harden visual publishing failure paths
Copilot Jul 23, 2026
93fe958
Count whole GitHub mention tokens in the merge budget guard (#36666 r…
kubaflo Jul 23, 2026
d45fef2
Preserve exact visual baseline provenance
Copilot Jul 24, 2026
ee579ba
Protect report parsing and gather finalization
Copilot Jul 24, 2026
e4b435a
Bound remaining gather enrichment
Copilot Jul 24, 2026
f8898d9
Enforce gather deadline across all requests
Copilot Jul 24, 2026
bf5b0fd
Bound gather subprocesses and evidence
Copilot Jul 24, 2026
284e34e
Preserve analysis around visual markers
Copilot Jul 24, 2026
d116698
Separate fresh and persisted visual markers
Copilot Jul 24, 2026
a122d99
Tighten AzDO attachment allowlist to reject adorned URLs (#36666 review)
kubaflo Jul 24, 2026
585e366
Neutralize untrusted visual marker content
Copilot Jul 24, 2026
3f81b62
Merge remote-tracking branch 'origin/review-tests-visual-comparisons'…
Copilot Jul 24, 2026
3f85eca
Document ruleset-compatible PAT environment policy
Copilot Jul 24, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .github/aw/actions-lock.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
{
"entries": {
"actions/checkout@v4": {
"repo": "actions/checkout",
"version": "v4",
"sha": "11d5960a326750d5838078e36cf38b85af677262"
},
"actions/checkout@v7.0.1": {
"repo": "actions/checkout",
"version": "v7.0.1",
Expand All @@ -25,11 +30,6 @@
"version": "v7.0.1",
"sha": "043fb46d1a93c77aae656e7c1c64a875d1fc6a0a"
},
"github/gh-aw-actions/setup-cli@v0.82.14": {
"repo": "github/gh-aw-actions/setup-cli",
"version": "v0.82.14",
"sha": "b6d1443e05b8716267fa19425b99aa4f12006b4a"
},
"github/gh-aw-actions/setup@v0.82.14": {
"repo": "github/gh-aw-actions/setup",
"version": "v0.82.14",
Expand Down
15 changes: 15 additions & 0 deletions .github/docs/maui-ci-facts.md
Original file line number Diff line number Diff line change
Expand Up @@ -268,6 +268,21 @@ Messages like `Baseline snapshot not yet created`, missing snapshot paths, or sn
environment-version mismatches are strong **unrelated** evidence — unless the PR adds or
modifies that visual test or the affected snapshot/platform.

The automated `/review tests` lane gathers failed UI result IDs from the public
`vstmr.dev.azure.com/.../testresults/resultsbybuild` endpoint, then reads the public
result-detail and attachment APIs. It publishes validated baseline/actual/diff PNGs to
the repository's `review-tests-assets` branch. A trusted post-step inserts as many
complete expandable comparison panels as fit inside the single test-failure analysis
comment while enforcing gh-aw's URL, mention, and character limits; excess panels are
reported as omitted rather than creating another comment. Visual publishing is
supplementary evidence only: missing images never raise or lower the deterministic
verdict ceiling. Each panel also shows a conservative relationship label derived from
the exact test-and-platform `deterministicAttribution` plus exact changed snapshot/test
scope: `regressed-vs-base` or directly changed visual coverage is Likely PR-caused,
`pre-existing-on-base` or `known-issue` is Likely unrelated, and indeterminate or
unmatched evidence remains Needs human investigation. A same-named snapshot on another
platform and platform or area mismatch alone never change the label.

## Platform mismatch

Platform mismatch is **supporting** evidence, not proof. An iOS-only test failing on a
Expand Down
Loading
Loading