[release/11.0.1xx-preview7] Bundle test-only stabilizations - #37081
Conversation
|
/azp run |
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 37081Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 37081" |
|
Azure Pipelines: Successfully started running 1 pipeline(s). There may be pipelines that require an authorized user to comment /azp run to run. |
|
Azure Pipelines: Successfully started running 3 pipeline(s). |
There was a problem hiding this comment.
Pull request overview
This PR bundles multiple UI test–only stabilizations for the release/11.0.1xx-preview7 branch, focusing on reducing UITest flakiness by adding deterministic waits/retries and slightly widening screenshot stabilization windows without changing product/runtime behavior.
Changes:
- Added/centralized more reliable Appium text-wait helpers (exact-text vs substring) and improved timeout diagnostics.
- Stabilized several flaky UI tests by waiting for keyboard/input-view transitions and retrying gestures/assertions where races occur.
- Adjusted
VerifyScreenshotusage to useretryTimeout(and in one case tolerance) to better accommodate UI settling in CI.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| src/TestUtils/src/UITest.Appium/HelperExtensions.cs | Adds shared text polling helper and introduces an exact-text wait API; improves timeout logging and corrects delay usage for this text wait loop. |
| src/Controls/tests/TestCases.Shared.Tests/Tests/Issues/Issue36154.cs | Waits for the SwipeView callback result label to update before asserting. |
| src/Controls/tests/TestCases.Shared.Tests/Tests/Issues/Issue28986_ParentChildTest.cs | Wraps initial SafeArea geometry assertions in RetryAssert to avoid reading transient layout values. |
| src/Controls/tests/TestCases.Shared.Tests/Tests/Issues/Issue28604.cs | Uses VerifyScreenshot retry window + small tolerance to reduce transient visual diffs. |
| src/Controls/tests/TestCases.Shared.Tests/Tests/Issues/Issue24496.cs | Adds iOS picker-wheel presence/absence waits and retries tapping Entry until focus is confirmed before snapshotting. |
| src/Controls/tests/TestCases.Shared.Tests/Tests/Issues/Issue18896.cs | Extends ListView screenshot stabilization window from 3s to 5s. |
| src/Controls/tests/TestCases.Shared.Tests/Tests/Issues/Issue16910.cs | Retries the pull-to-refresh gesture once if the initial refresh state doesn’t appear quickly. |
| src/Controls/tests/TestCases.Shared.Tests/Tests/FeatureMatrix/EntryFeatureTests.cs | On iOS Appium runs, waits for the keyboard to show before taking the focused-entry snapshot. |
| src/Controls/tests/TestCases.Shared.Tests/Tests/DragAndDropUITests.cs | Uses exact-text polling for drag/drop event label assertions to avoid placeholder substring races. |
|
Expanded this aggregate beyond UI tests: it now includes #37044's independently valid Windows test-infrastructure fallback and its 14 focused Pester cases. Device tests, other test types, infrastructure, and screenshots are explicitly in scope when they pass without product-code changes. /azp run |
|
/azp run |
|
Azure Pipelines: Successfully started running 3 pipeline(s). |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.
Suppressed comments (2)
eng/scripts/Set-ScreenResolution.ps1:58
- Test-ScreenResolutionApplySucceeded hard-codes success/restart values (0/1). Consider using the named DISP_CHANGE_* constants (with a fallback when DisplaySettings.NativeMethods isn't loaded) to avoid duplicating Win32 return codes.
function Test-ScreenResolutionApplySucceeded {
param (
[int]$Result
)
return ($Result -eq 0 -or $Result -eq 1)
}
eng/scripts/Set-ScreenResolution.ps1:50
- Get-ScreenResolutionProbeAction hard-codes the BADMODE value (-2). Since the script already defines the native constants in DisplaySettings.NativeMethods, using the named constant (with a fallback when the type isn't loaded) would reduce magic numbers and keep the intent clear.
This issue also appears on line 52 of the same file.
function Get-ScreenResolutionProbeAction {
param (
[int]$Result
)
if ($Result -eq -2) {
return "Reject"
}
return "Apply"
}
|
/azp run |
|
Azure Pipelines: Successfully started running 3 pipeline(s). |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.
Suppressed comments (1)
src/Controls/tests/TestCases.Shared.Tests/Tests/DragAndDropUITests.cs:71
- WaitForTextEqualToElement returns false on timeout (it does not throw), but this call ignores the return value and then proceeds to read/assert the label text. If the wait times out, the failure message becomes less actionable and the method continues with an element text read that can reintroduce transient Appium flakiness. Assert the return value so timeouts fail deterministically with a clear reason (and apply the same pattern to the other WaitForTextEqualToElement calls in this test).
App.WaitForTextEqualToElement("DragStartEventsLabel", "DragStarting");
PureWeen
left a comment
There was a problem hiding this comment.
Adversarial code review
Code/test-correctness verdict: READY TO MERGE
One non-blocking warning survived adversarial consensus:
⚠️ Testing — The new iOS keyboard-readiness guard is applied to oneTap("TestEntry")→VerifyScreenshotWithKeyboardHandling()path, while several equivalent active call sites still snapshot without waiting for keyboard readiness. This does not regress the patched test, but leaves the same timing failure possible elsewhere.
No must-fix correctness defects survived consensus. The suspected Graphics dependency on #37057’s excluded product change was rejected after verifying that the current ScaleImage implementation is already main-thread-independent. The Pester suite correctly covers the changed decision seams; its broader native flow remains outside this Linux-runnable unit scope.
Coverage assessment: the directly reused file patches match their source PR heads, the Graphics extraction omits the four nonpositive-size cases that require the product fix, and the changed UI/device/infrastructure paths have targeted tests. Existing automated reviews contained no unresolved actionable inline findings duplicated here.
Methodology: 3 independent reviewers with adversarial consensus + repo domain specialist. CI status is intentionally evaluated separately from this code review.
| App.WaitForElement("TestEntry"); | ||
| App.Tap("TestEntry"); | ||
| if (App is AppiumIOSApp) | ||
| Assert.That(App.WaitForKeyboardToShow(), Is.True, "The iOS keyboard did not appear before the snapshot."); |
There was a problem hiding this comment.
Tap("TestEntry") → VerifyScreenshotWithKeyboardHandling() sequence and the shared helper does not wait for keyboard readiness. On a slower iOS transition, those tests can still capture before the keyboard settles. Consider applying the guard to the equivalent iOS call sites or making keyboard readiness an explicit opt-in of the shared helper.
Flagged by: 1/3 reviewers + repo specialist after dispute
There was a problem hiding this comment.
Thanks — agreed this is a remaining coverage opportunity. I kept this aggregate scoped to the exact Entry path that failed in Preview 7. The shared helper is also used by cases where the iOS keyboard may intentionally be absent or transition differently, so making the wait unconditional could create new failures. I am treating this as non-blocking and will add the guard to another specific call site if exact CI evidence shows the same race there.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.
Suppressed comments (1)
src/Controls/tests/TestCases.Shared.Tests/Tests/Issues/Issue36154.cs:34
- The comment says the test would pass if the SwipeView invoke callback never updates the label, but the prior assertion would have failed in that case. The main value here is waiting for the callback to complete (reducing flakiness) and asserting on the boolean timeout result.
// WaitForTextToBePresentInElement returns false (rather than throwing) on timeout, so assert on
// it: otherwise the test would pass even if the SwipeView invoke callback never updated the label.
PureWeen
left a comment
There was a problem hiding this comment.
Deep MAUI test/infrastructure review at 67525bdd55fcbc9012a4db2660d35661ed69a208. Two inline findings survived reconciliation with the existing review threads. maui-pr and maui-pr-uitests are still pending; exact-head device tests are green.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9be49656-7117-4235-9d96-404779ab6b16
|
/azp run |
|
Azure Pipelines: Successfully started running 3 pipeline(s). |
Note
Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!
Description of Change
Bundles independently valid Preview 7 test and test-infrastructure stabilizations into one merge path. Changes are cherry-picked or extracted from their source PRs without modifying MAUI runtime or product behavior.
The aggregate currently covers:
Eligible UI tests, device tests, unit tests, test infrastructure, and screenshot baselines can all be included here when they are independently valid without a corresponding product-code change.
Source PRs
The standalone source PRs below are closed in favor of this aggregate. For #37057, only the independently valid test subset is included; its product-code rewrite remains excluded.
Scope exclusions
Mixed runtime/test fixes are included only when their test-side changes pass independently against the unmodified product code. The current device tests in #37052, the four nonpositive-size cases in #37057, and the device test plus screenshots in #37062 remain excluded because they expose or describe behavior that requires those PRs' functional fixes. #37070 has no test-side changes. Closed ineffective or unsafe fixes are also excluded.
Validation
UITest.Appium,Controls.TestCases.iOS.Tests,Controls.TestCases.Mac.Tests, andControls.TestCases.Android.Tests.DisplayAlertAsyncreturns, avoiding a teardown-order dependency on the detached page task.dee83edd121; the resulting diff remains limited to tests and test infrastructure.eng/scripts/Set-ScreenResolution.ps1changes./azp runwas posted for aggregate head83da465424b; exact mergebad9e4e9a57builds1539860,1539861, and1539862are required before merge.Issues Fixed
Contributes to stabilizing the .NET 11 Preview 7 test branch.