Fix flaky UI tests with retryTimeout, crash recovery, and improved diagnostics by PureWeen · Pull Request #33705 · dotnet/maui

PureWeen · 2026-01-25T23:23:47Z

Note

Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!

Description

This PR addresses multiple sources of UI test flakiness identified from analyzing the last 10 main branch UITest CI runs.

1. Screenshot Timing Fixes (`retryTimeout` parameter)

Many visual tests were failing intermittently because screenshots were taken before animations/transitions completed. Added retryTimeout: TimeSpan.FromSeconds(2) to tests that tap buttons and immediately verify screenshots:

FeatureMatrix tests (39 files):

All BoxView, Border, Brush, Button, CarouselView, CollectionView, ContentView, Frame, GraphicsView, Image, ImageButton, Label, ScrollView, Shadow, Shapes, Stepper, SwipeView, Switch, WebView tests

Issue tests:

Issue10563 (SwipeView open/close)
Issue27730 (Shadow update when clipping)
Issue26662 (Dynamic FontImageSource color)

2. Instrumentation Crash Recovery

Added detection and recovery for Android instrumentation crashes that were causing entire test fixtures to fail:

Added IsInstrumentationCrash() method with 12 crash signatures (socket hang up, ECONNRESET, session terminated, etc.)
Added recovery in FixtureSetup and TestSetup that calls base.Reset() to recreate the Appium driver session
This allows tests to recover from transient instrumentation failures instead of failing the entire fixture

3. OneTimeSetUp Diagnostic Attachment Fix

Fixed an issue where diagnostic files (logcat, screenshots) attached during OneTimeSetUp failures were not visible in Azure DevOps test results:

NUnit limitation: TestContext.AddTestAttachment in OneTimeSetUp attaches to fixture context, not individual tests
Azure DevOps only displays per-test attachments
Solution: Store diagnostic file paths during fixture setup failures, then re-attach them to each individual test in TearDown

Changes

Infrastructure (`src/TestUtils/src/UITest.NUnit/UITestBase.cs`)

Added _fixtureSetupDiagnosticFiles list and _fixtureSetupFailed flag
Modified TearDown to re-attach fixture diagnostic files to each test
Added storeForReattachment parameter to SaveDeviceDiagnosticInfo and SaveUIDiagnosticInfo

MAUI Test Base (`src/Controls/tests/TestCases.Shared.Tests/UITest.cs`)

Added IsInstrumentationCrash() detection with 12 crash signatures
Added crash recovery in FixtureSetup and TestSetup

Test Files

39 FeatureMatrix test files: Added retryTimeout
Issue10563.cs: Added retryTimeout to SwipeView screenshot tests
Issue27730.cs: Added retryTimeout to shadow update test
Issue26662.cs: Added retryTimeout to dynamic color test

Testing

Validated through multiple CI runs:

Build 1270034: iOS/macOS all passed, Android CoreClr passed after retry
Build 1270368: Identified additional timing issues in Issue27730 and Issue26662
Build 1270512: In progress with latest fixes

PureWeen · 2026-01-25T23:23:53Z

/azp run maui-pr-uitests

azure-pipelines · 2026-01-25T23:24:03Z

Azure Pipelines successfully started running 1 pipeline(s).

PureWeen · 2026-01-25T23:27:20Z

/azp run maui-pr-uitests

azure-pipelines · 2026-01-25T23:27:28Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot

Pull request overview

This PR refines several flaky .NET MAUI UI tests by improving synchronization around animations/scrolling and making the visual-diff helper more robust to localized percentage formats.

Changes:

Generalized the visual-diff percentage parsing in UITest.cs to handle both dot and comma decimal separators.
Hardened Issue10563, Issue24034, Issue24996, and multiple BoxView/ScrollView feature tests with targeted waits and screenshot tolerances to avoid timing-related flakiness.
Increased resilience of Issue24996 by re-querying UI elements after interactions to avoid stale element references.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`src/Controls/tests/TestCases.Shared.Tests/UITest.cs`	Adjusts the regex used in `ExtractDifferencePercentage` so it parses both `X.XX% difference` and `X,XX% difference`, avoiding locale-specific parsing issues.
`src/Controls/tests/TestCases.Shared.Tests/Tests/Issues/Issue24996.cs`	Updates the translation test to tap by automation ID, wait longer for animations, and re-query the “Stats” element each iteration before assertions to avoid stale references.
`src/Controls/tests/TestCases.Shared.Tests/Tests/Issues/Issue24034.cs`	Adds a 300 ms delay and a 3% screenshot tolerance after toggling the button so the shadow animation can complete before visual comparison.
`src/Controls/tests/TestCases.Shared.Tests/Tests/Issues/Issue10563.cs`	Makes the SwipeView open/close test deterministic by waiting for the SwipeItem text to appear/disappear around each screenshot, though the bottom-close path currently omits the final `WaitForNoElement`.
`src/Controls/tests/TestCases.Shared.Tests/Tests/FeatureMatrix/ScrollViewFeatureTests.cs`	Adds short blocking delays and 2% screenshot tolerance to five ScrollView scenarios so screenshots are taken after scroll/RTL layout animations have settled.
`src/Controls/tests/TestCases.Shared.Tests/Tests/FeatureMatrix/BoxViewFeatureTests.cs`	Adds 300 ms delays and 2% screenshot tolerance to seven BoxView feature tests to let visibility, color, opacity, flow direction, and shadow changes stabilize before snapshot comparison.

Copilot · 2026-01-25T23:33:30Z

src/Controls/tests/TestCases.Shared.Tests/Tests/Issues/Issue10563.cs

 			App.Tap(OpenBottomId);
+			App.WaitForElement("Issue 10563");
 			VerifyScreenshotOrSetException(ref exception, "Bottom_SwipeItems");
 			App.Tap(CloseId);


For consistency and to ensure the SwipeView has fully closed in all cases, it would be better to also wait for the "Issue 10563" text to disappear after tapping CloseId for the bottom SwipeItems, similar to the left/right/top cases above. This keeps the test deterministic and avoids leaving a potentially still-animating SwipeView when the test completes.

Suggested change

App.Tap(CloseId);

App.Tap(CloseId);

App.WaitForNoElement("Issue 10563");

PureWeen · 2026-01-26T01:50:27Z

/azp run maui-pr-uitests

azure-pipelines · 2026-01-26T01:50:38Z

Azure Pipelines successfully started running 1 pipeline(s).

PureWeen · 2026-01-26T02:38:36Z

/azp run maui-pr-uitests

azure-pipelines · 2026-01-26T02:38:46Z

Azure Pipelines successfully started running 1 pipeline(s).

PureWeen · 2026-01-26T02:41:42Z

/azp run maui-pr-uitests

azure-pipelines · 2026-01-26T02:41:51Z

Azure Pipelines successfully started running 1 pipeline(s).

PureWeen · 2026-01-26T18:42:59Z

/azp run maui-pr-uitests

azure-pipelines · 2026-01-26T18:43:09Z

Azure Pipelines successfully started running 1 pipeline(s).

PureWeen · 2026-01-26T20:04:27Z

/azp run maui-pr-uitests

azure-pipelines · 2026-01-26T20:04:38Z

Azure Pipelines successfully started running 1 pipeline(s).

PureWeen · 2026-01-26T21:10:17Z

/azp run maui-pr-uitests

azure-pipelines · 2026-01-26T21:10:28Z

Azure Pipelines successfully started running 1 pipeline(s).

PureWeen · 2026-01-26T21:18:27Z

/azp run maui-pr-uitests

azure-pipelines · 2026-01-26T21:18:36Z

Azure Pipelines successfully started running 1 pipeline(s).

PureWeen · 2026-01-26T22:29:57Z

/azp run maui-pr-uitests

azure-pipelines · 2026-01-26T22:30:08Z

Azure Pipelines successfully started running 1 pipeline(s).

PureWeen · 2026-01-27T01:32:14Z

/azp run maui-pr-uitests

azure-pipelines · 2026-01-27T01:32:25Z

Azure Pipelines successfully started running 1 pipeline(s).

- Issue10563: Replace Task.Delay with WaitForNoElement after Close() - ScrollViewFeatureTests: Add 300ms delay and 2% tolerance for screenshots

- BoxViewFeatureTests: Add 300ms delay and 2% tolerance to all 7 tests - UITest.cs: Fix regex to match both comma and dot decimal separators

- Issue24034: Add 300ms delay and 3% tolerance for shadow animation - Issue24996: Increase delay to 300ms, re-query element to avoid stale reference

- Issue32394: Add 500ms delay after orientation change, 2% tolerance - Issue29109: Add 300ms delay and tolerance (2% for static, 4% for dynamic) - Issue22306: Add delays after orientation changes and 2% tolerance to all screenshots

This commit introduces a retryTimeout parameter to VerifyScreenshot() that allows tests to keep retrying until the screenshot matches, replacing arbitrary Task.Delay() calls with a more robust approach. Key changes: - Add retryTimeout parameter to VerifyScreenshot() and VerifyScreenshotOrSetException() - When retryTimeout is specified, retry every retryDelay (default 500ms) until timeout - Replace Task.Delay() with retryTimeout in flaky tests: - BoxViewFeatureTests (7 tests) - ScrollViewFeatureTests (5 tests) - Issue10563, Issue22306, Issue24034, Issue29109, Issue32394 - Reduce tolerance from 2-4% to 0.5% - retryTimeout handles timing, small tolerance provides safety margin for CI rendering differences - Add flaky test investigation guidance to uitests.instructions.md Benefits: - More robust: keeps retrying until UI settles, rather than guessing delay - Safer: small tolerance provides CI safety margin without hiding real bugs - Cleaner: single retryTimeout parameter instead of Task.Delay() calls

Apply the same fix pattern used for BoxView and ScrollView tests: - Add retryTimeout: TimeSpan.FromSeconds(2) to handle timing variance - Add tolerance: 0.5 for cross-machine rendering variance Also update write-tests skill documentation with: - Platform selection guidance - retryTimeout best practices - Dynamic platform parameters - XAML optional note

Apply retryTimeout pattern to fix flaky screenshot tests: - ContentViewFeatureTests.cs - AppThemeFeatureTests.cs - Issue24489_Shell.cs - Issue24489_2.cs

Comprehensive update to fix flaky screenshot tests across all FeatureMatrix tests. The most consistently flaky category on main was SoftInput,Stepper,Switch,SwipeView (failed 4/4 recent builds). Updated 39 files with: - tolerance: 0.5 for cross-machine rendering variance - retryTimeout: TimeSpan.FromSeconds(2) for timing variance

…eview

…anges

- ChangePeekAreaInsetsInOnSizeAllocatedTest: Add 30s timeout for CarouselId element - RemoveItemsQuickly: Add 30s timeout for initial element, 10s for post-action verification - Issue12574Test: Add timeouts for all carousel item waits These tests are flaky on CI due to CarouselView rendering time variance.

When Android UiAutomator2 crashes mid-test (e.g., 'socket hang up', 'instrumentation process is not running', 'Can't find service: package'), the test infrastructure now detects this and attempts to recreate the Appium session by calling base.Reset() which disposes and recreates the driver. This prevents cascading test failures where all subsequent tests fail because the automation infrastructure is dead. Recovery is added in two places: - FixtureSetup: Detects crash during test class initialization - TestSetup: Detects crash during SetOrientationPortrait() The detection covers 12 known crash signatures based on CI log analysis.

NUnit's TestContext.AddTestAttachment in OneTimeSetUp/OneTimeTearDown attaches files to the fixture context, not individual test results. Azure DevOps only displays per-test attachments, so logcat files from fixture setup failures were not visible. This fix stores the diagnostic file paths when captured during OneTimeSetUp failure, then re-attaches them to each individual test in TearDown so they appear in Azure DevOps test results.

rmarinho · 2026-01-28T11:44:01Z

/azp run maui-pr-uitests

azure-pipelines · 2026-01-28T11:44:13Z

Azure Pipelines successfully started running 1 pipeline(s).

…agnostics (#33705)  > [!NOTE] > Are you waiting for the changes in this PR to be merged? > It would be very helpful if you could [test the resulting artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from this PR and let us know in a comment if this change resolves your issue. Thank you! ## Description This PR addresses multiple sources of UI test flakiness identified from analyzing the last 10 main branch UITest CI runs. ### 1. Screenshot Timing Fixes (`retryTimeout` parameter) Many visual tests were failing intermittently because screenshots were taken before animations/transitions completed. Added `retryTimeout: TimeSpan.FromSeconds(2)` to tests that tap buttons and immediately verify screenshots: **FeatureMatrix tests (39 files):** - All BoxView, Border, Brush, Button, CarouselView, CollectionView, ContentView, Frame, GraphicsView, Image, ImageButton, Label, ScrollView, Shadow, Shapes, Stepper, SwipeView, Switch, WebView tests **Issue tests:** - Issue10563 (SwipeView open/close) - Issue27730 (Shadow update when clipping) - Issue26662 (Dynamic FontImageSource color) ### 2. Instrumentation Crash Recovery Added detection and recovery for Android instrumentation crashes that were causing entire test fixtures to fail: - Added `IsInstrumentationCrash()` method with 12 crash signatures (socket hang up, ECONNRESET, session terminated, etc.) - Added recovery in `FixtureSetup` and `TestSetup` that calls `base.Reset()` to recreate the Appium driver session - This allows tests to recover from transient instrumentation failures instead of failing the entire fixture ### 3. OneTimeSetUp Diagnostic Attachment Fix Fixed an issue where diagnostic files (logcat, screenshots) attached during `OneTimeSetUp` failures were not visible in Azure DevOps test results: - NUnit limitation: `TestContext.AddTestAttachment` in `OneTimeSetUp` attaches to fixture context, not individual tests - Azure DevOps only displays per-test attachments - Solution: Store diagnostic file paths during fixture setup failures, then re-attach them to each individual test in `TearDown` ## Changes ### Infrastructure (`src/TestUtils/src/UITest.NUnit/UITestBase.cs`) - Added `_fixtureSetupDiagnosticFiles` list and `_fixtureSetupFailed` flag - Modified `TearDown` to re-attach fixture diagnostic files to each test - Added `storeForReattachment` parameter to `SaveDeviceDiagnosticInfo` and `SaveUIDiagnosticInfo` ### MAUI Test Base (`src/Controls/tests/TestCases.Shared.Tests/UITest.cs`) - Added `IsInstrumentationCrash()` detection with 12 crash signatures - Added crash recovery in `FixtureSetup` and `TestSetup` ### Test Files - 39 FeatureMatrix test files: Added `retryTimeout` - Issue10563.cs: Added `retryTimeout` to SwipeView screenshot tests - Issue27730.cs: Added `retryTimeout` to shadow update test - Issue26662.cs: Added `retryTimeout` to dynamic color test ## Testing Validated through multiple CI runs: - Build 1270034: iOS/macOS all passed, Android CoreClr passed after retry - Build 1270368: Identified additional timing issues in Issue27730 and Issue26662 - Build 1270512: In progress with latest fixes

…agnostics (dotnet#33705)  > [!NOTE] > Are you waiting for the changes in this PR to be merged? > It would be very helpful if you could [test the resulting artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from this PR and let us know in a comment if this change resolves your issue. Thank you! ## Description This PR addresses multiple sources of UI test flakiness identified from analyzing the last 10 main branch UITest CI runs. ### 1. Screenshot Timing Fixes (`retryTimeout` parameter) Many visual tests were failing intermittently because screenshots were taken before animations/transitions completed. Added `retryTimeout: TimeSpan.FromSeconds(2)` to tests that tap buttons and immediately verify screenshots: **FeatureMatrix tests (39 files):** - All BoxView, Border, Brush, Button, CarouselView, CollectionView, ContentView, Frame, GraphicsView, Image, ImageButton, Label, ScrollView, Shadow, Shapes, Stepper, SwipeView, Switch, WebView tests **Issue tests:** - Issue10563 (SwipeView open/close) - Issue27730 (Shadow update when clipping) - Issue26662 (Dynamic FontImageSource color) ### 2. Instrumentation Crash Recovery Added detection and recovery for Android instrumentation crashes that were causing entire test fixtures to fail: - Added `IsInstrumentationCrash()` method with 12 crash signatures (socket hang up, ECONNRESET, session terminated, etc.) - Added recovery in `FixtureSetup` and `TestSetup` that calls `base.Reset()` to recreate the Appium driver session - This allows tests to recover from transient instrumentation failures instead of failing the entire fixture ### 3. OneTimeSetUp Diagnostic Attachment Fix Fixed an issue where diagnostic files (logcat, screenshots) attached during `OneTimeSetUp` failures were not visible in Azure DevOps test results: - NUnit limitation: `TestContext.AddTestAttachment` in `OneTimeSetUp` attaches to fixture context, not individual tests - Azure DevOps only displays per-test attachments - Solution: Store diagnostic file paths during fixture setup failures, then re-attach them to each individual test in `TearDown` ## Changes ### Infrastructure (`src/TestUtils/src/UITest.NUnit/UITestBase.cs`) - Added `_fixtureSetupDiagnosticFiles` list and `_fixtureSetupFailed` flag - Modified `TearDown` to re-attach fixture diagnostic files to each test - Added `storeForReattachment` parameter to `SaveDeviceDiagnosticInfo` and `SaveUIDiagnosticInfo` ### MAUI Test Base (`src/Controls/tests/TestCases.Shared.Tests/UITest.cs`) - Added `IsInstrumentationCrash()` detection with 12 crash signatures - Added crash recovery in `FixtureSetup` and `TestSetup` ### Test Files - 39 FeatureMatrix test files: Added `retryTimeout` - Issue10563.cs: Added `retryTimeout` to SwipeView screenshot tests - Issue27730.cs: Added `retryTimeout` to shadow update test - Issue26662.cs: Added `retryTimeout` to dynamic color test ## Testing Validated through multiple CI runs: - Build 1270034: iOS/macOS all passed, Android CoreClr passed after retry - Build 1270368: Identified additional timing issues in Issue27730 and Issue26662 - Build 1270512: In progress with latest fixes

Copilot AI review requested due to automatic review settings January 25, 2026 23:23

Copilot started reviewing on behalf of PureWeen January 25, 2026 23:24 View session

Copilot AI reviewed Jan 25, 2026

View reviewed changes

PureWeen marked this pull request as draft January 26, 2026 00:00

build-analysis bot mentioned this pull request Jan 26, 2026

System.TimeoutException #33245

Open

PureWeen force-pushed the fix-main branch 2 times, most recently from 474aa4e to ec65bee Compare January 26, 2026 19:44

PureWeen marked this pull request as ready for review January 27, 2026 01:28

PureWeen added 13 commits January 27, 2026 13:05

Fix flaky UI tests: SwipeView and ScrollView timing issues

ea79f27

- Issue10563: Replace Task.Delay with WaitForNoElement after Close() - ScrollViewFeatureTests: Add 300ms delay and 2% tolerance for screenshots

Fix flaky BoxView tests and tolerance regex bug

1301719

- BoxViewFeatureTests: Add 300ms delay and 2% tolerance to all 7 tests - UITest.cs: Fix regex to match both comma and dot decimal separators

Fix flaky ShadowShouldUpdate and ChangingTranslation tests

ad38b6d

- Issue24034: Add 300ms delay and 3% tolerance for shadow animation - Issue24996: Increase delay to 300ms, re-query element to avoid stale reference

Fix additional flaky screenshot tests

a919d32

- Issue32394: Add 500ms delay after orientation change, 2% tolerance - Issue29109: Add 300ms delay and tolerance (2% for static, 4% for dynamic) - Issue22306: Add delays after orientation changes and 2% tolerance to all screenshots

Add 4% tolerance to Issue10563 SwipeView screenshot tests

0030bac

Add retryTimeout to ViewBaseTests and Window tests

f620c9d

Apply retryTimeout pattern to fix flaky screenshot tests: - ContentViewFeatureTests.cs - AppThemeFeatureTests.cs - Issue24489_Shell.cs - Issue24489_2.cs

Remove Task.Delay anti-pattern from Issue24996 based on multi-model r…

bcaab4c

…eview

Increase Issue32394 tolerance to 1.5% for CarouselView orientation ch…

d070abd

…anges

PureWeen force-pushed the fix-main branch from 038bc9e to 19d6c20 Compare January 27, 2026 19:07

PureWeen added 2 commits January 27, 2026 14:30

Add retryTimeout to Issue10563 SwipeView screenshot tests

d94c8f9

Add retryTimeout to Issue27730 and Issue26662 visual tests

199e29f

PureWeen changed the title ~~Fix flaky UI tests: timing and screenshot tolerance issues~~ Fix flaky UI tests with retryTimeout, crash recovery, and improved diagnostics Jan 28, 2026

rmarinho approved these changes Jan 28, 2026

View reviewed changes

rmarinho merged commit 7299684 into main Jan 28, 2026
136 of 149 checks passed

rmarinho deleted the fix-main branch January 28, 2026 16:49

build-analysis bot mentioned this pull request Jan 28, 2026

Project build failure causing test run abort in RunOniOS_BlazorRelease on mac_runios_blazor_release job #33744

Closed

PureWeen mentioned this pull request Jan 28, 2026

[Testing] Fix flaky UI tests: retryTimeout and SwipeView button fix #33749

Merged

	App.Tap(CloseId);
	App.Tap(CloseId);
	App.WaitForNoElement("Issue 10563");

Conversation

PureWeen commented Jan 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

1. Screenshot Timing Fixes (retryTimeout parameter)

2. Instrumentation Crash Recovery

3. OneTimeSetUp Diagnostic Attachment Fix

Changes

Infrastructure (src/TestUtils/src/UITest.NUnit/UITestBase.cs)

MAUI Test Base (src/Controls/tests/TestCases.Shared.Tests/UITest.cs)

Test Files

Testing

Uh oh!

PureWeen commented Jan 25, 2026

Uh oh!

azure-pipelines bot commented Jan 25, 2026

Uh oh!

PureWeen commented Jan 25, 2026

Uh oh!

azure-pipelines bot commented Jan 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

PureWeen commented Jan 26, 2026

Uh oh!

azure-pipelines bot commented Jan 26, 2026

Uh oh!

PureWeen commented Jan 26, 2026

Uh oh!

azure-pipelines bot commented Jan 26, 2026

Uh oh!

PureWeen commented Jan 26, 2026

Uh oh!

azure-pipelines bot commented Jan 26, 2026

Uh oh!

PureWeen commented Jan 26, 2026

Uh oh!

azure-pipelines bot commented Jan 26, 2026

Uh oh!

PureWeen commented Jan 26, 2026

Uh oh!

azure-pipelines bot commented Jan 26, 2026

Uh oh!

PureWeen commented Jan 26, 2026

Uh oh!

azure-pipelines bot commented Jan 26, 2026

Uh oh!

PureWeen commented Jan 26, 2026

Uh oh!

azure-pipelines bot commented Jan 26, 2026

Uh oh!

PureWeen commented Jan 26, 2026

Uh oh!

azure-pipelines bot commented Jan 26, 2026

Uh oh!

PureWeen commented Jan 27, 2026

Uh oh!

azure-pipelines bot commented Jan 27, 2026

Uh oh!

rmarinho commented Jan 28, 2026

Uh oh!

azure-pipelines bot commented Jan 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

PureWeen commented Jan 25, 2026 •

edited

Loading

1. Screenshot Timing Fixes (`retryTimeout` parameter)

Infrastructure (`src/TestUtils/src/UITest.NUnit/UITestBase.cs`)

MAUI Test Base (`src/Controls/tests/TestCases.Shared.Tests/UITest.cs`)