Split iOS device tests by category in Helix #33263
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add category-based splitting for iOS Controls.DeviceTests and Core.DeviceTests - Each category runs as a separate Helix work item (~95 total for iOS) - Uses CustomCommands with --set-env="TestFilter=Category=X" like old cake approach - MacCatalyst and Android unchanged (still run as single work items per project) - Keep category lists in sync with TestCategory.cs files
- Only split Controls.DeviceTests (not Core.DeviceTests) - Only 7 heavy categories run separately: CollectionView, CarouselView, FlyoutPage, Shell, TabbedPage, WebView, HybridWebView - All other Controls tests run as one 'Other' work item - Core.DeviceTests runs as single work item - Reduces iOS work items from ~95 to ~12
- Add SkipCategories environment variable support in DeviceTestSharedHelpers - SkipCategories takes comma-separated list of categories to skip - Define ControlsTestCategoriesToSkipForRestOfTests property once, reuse for both splitting and skipping - Heavy categories run individually: CollectionView, CarouselView, FlyoutPage, Shell, TabbedPage, WebView, HybridWebView
- Add TestFilter=SkipCategories=X,Y,Z support in DeviceTestSharedHelpers - Define ControlsTestCategoriesToSkipForRestOfTests property for heavy categories - Heavy categories run individually: CollectionView, CarouselView, FlyoutPage, Shell, TabbedPage, WebView, HybridWebView - All other Controls tests run in a single 'Other' work item - Core.DeviceTests runs as single work item (no splitting)
Member
Author
|
/rebase |
Member
Author
|
/azp run |
|
Azure Pipelines successfully started running 3 pipeline(s). |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR optimizes iOS device test execution on Helix by splitting heavy test categories into separate parallel work items, reducing overall test time. The implementation adds support for skipping multiple test categories and provides comprehensive documentation for Helix device test configuration.
Key Changes
- Splits iOS Controls.DeviceTests into 8 parallel work items (7 heavy categories + 1 "Other")
- Adds
SkipCategoriesfilter support to enable running all tests except specified categories - Introduces detailed Copilot instructions for Helix device test configuration and troubleshooting
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
src/Core/tests/DeviceTests.Shared/DeviceTestSharedHelpers.cs |
Adds SkipCategories=X,Y,Z filter support with comma/semicolon-separated category list parsing |
eng/helix_xharness.proj |
Implements iOS-specific category splitting with 7 heavy categories as individual work items and remaining categories in "Other" work item |
.github/instructions/helix-device-tests.instructions.md |
New documentation covering Helix configuration, iOS category splitting, local execution, and troubleshooting guidance |
This was referenced Dec 23, 2025
- Filter empty strings from SkipCategories split to handle trailing separators - Update documentation to reference correct property/ItemGroup names - Update heavy categories list to match current implementation
3 tasks
Member
Author
|
Currently our DeviceTest runs are taking over an hour and timing out. This fixes it so they all take under 9 minutes |
3 tasks
StephaneDelcroix
pushed a commit
that referenced
this pull request
Jan 5, 2026
<!-- Please let the below note in for people that find this PR --> > [!NOTE] > Are you waiting for the changes in this PR to be merged? > It would be very helpful if you could [test the resulting artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from this PR and let us know in a comment if this change resolves your issue. Thank you! ## Description Split iOS device tests by category in Helix to work around an unknown issue that causes tests to take over an hour when run together, vs under 15 minutes when split. ## Problem When running all Controls.DeviceTests categories together on iOS in CI, the test run takes over an hour. When the same tests are split by category, they complete in under 15 minutes total. This issue only reproduces in CI - we haven't been able to reproduce it locally. Ideally we'd find and fix the root cause, but until then this workaround keeps CI times reasonable. ## Changes ### Helix Configuration (`eng/helix_xharness.proj`) - Add `ControlsTestCategoriesToSkipForRestOfTests` property defining heavy categories to run individually - Split Controls.DeviceTests into 4 work items for iOS: - 3 heavy categories run separately: `CollectionView`, `Shell`, `HybridWebView` - 1 "Other" work item runs all remaining categories - Core.DeviceTests runs as a single work item (no splitting) - MacCatalyst and Android unchanged ### Test Infrastructure (`src/Core/tests/DeviceTests.Shared/DeviceTestSharedHelpers.cs`) - Add support for `TestFilter=SkipCategories=X,Y,Z` environment variable - Allows skipping multiple categories via comma or semicolon-separated list - Existing `TestFilter=Category=X` support unchanged ### Documentation (`.github/instructions/helix-device-tests.instructions.md`) - Add Copilot instructions for Helix device test configuration - Document category splitting, local execution, and troubleshooting ## Result iOS device tests now run as ~8 parallel work items instead of 5, with the slowest categories isolated. This brings iOS device test time from 1+ hour down to ~15 minutes.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note
Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!
Description
Split iOS device tests by category in Helix to work around an unknown issue that causes tests to take over an hour when run together, vs under 15 minutes when split.
Problem
When running all Controls.DeviceTests categories together on iOS in CI, the test run takes over an hour. When the same tests are split by category, they complete in under 15 minutes total. This issue only reproduces in CI - we haven't been able to reproduce it locally. Ideally we'd find and fix the root cause, but until then this workaround keeps CI times reasonable.
Changes
Helix Configuration (
eng/helix_xharness.proj)ControlsTestCategoriesToSkipForRestOfTestsproperty defining heavy categories to run individuallyCollectionView,Shell,HybridWebViewTest Infrastructure (
src/Core/tests/DeviceTests.Shared/DeviceTestSharedHelpers.cs)TestFilter=SkipCategories=X,Y,Zenvironment variableTestFilter=Category=Xsupport unchangedDocumentation (
.github/instructions/helix-device-tests.instructions.md)Result
iOS device tests now run as ~8 parallel work items instead of 5, with the slowest categories isolated. This brings iOS device test time from 1+ hour down to ~15 minutes.