Skip to content

[ai] Skill for running device tests #33484

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rmarinho merged 7 commits into from
Jan 28, 2026
+1,007 −24
Merged

[ai] Skill for running device tests #33484

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
ca60d45
Commit the skill to run xharnes ios
rmarinho Jan 12, 2026
92dcc65
Refactor run-device-tests skill to reuse existing infrastructure
PureWeen Jan 17, 2026
2430995
Improve XHarness device and iOS version targeting
kubaflo Jan 21, 2026
6b54399
Update skill to run different platforms iOS,ANDroid
rmarinho Jan 27, 2026
50fbba3
Run Android with filters and witth skill
rmarinho Jan 27, 2026
fe1b8e8
Fix DeviceTestSharedHelprs
rmarinho Jan 27, 2026
5e4c24a
Update skill information
rmarinho Jan 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions .github/instructions/helix-device-tests.instructions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,26 @@ dotnet tool restore
./build.sh -restore -build -configuration Release /p:BuildDeviceTests=true /bl:BuildDeviceTests.binlog -warnAsError false
```

### Using the run-device-tests Skill (Recommended)

The easiest way to run device tests locally is using the `run-device-tests` skill:

```bash
# Run Controls tests on iOS simulator
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform ios

# Run only Button category tests
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform ios -TestFilter "Category=Button"

# Run on Android emulator
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform android -TestFilter "Category=Button"

# Run on MacCatalyst
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Core -Platform maccatalyst
```

See `.github/skills/run-device-tests/SKILL.md` for full documentation.

### Submit to Helix

Set required environment variables:
Expand Down Expand Up @@ -118,6 +138,37 @@ To validate the helix proj without submitting (requires built artifacts):
dotnet msbuild eng/helix_xharness.proj /t:DiscoverTestBundles /p:TargetOS=ios /p:_MauiDotNetTfm=net10.0 /p:RepoRoot=$(pwd)/ -v:n
```

## Test Filtering Implementation

Test category filtering is implemented in `src/Core/tests/DeviceTests.Shared/DeviceTestSharedHelpers.cs`. The `GetExcludedTestCategories()` method reads the `TestFilter` value and converts it to a list of categories to skip.

### Filter Syntax

| Format | Description | Example |
|--------|-------------|---------|
| `Category=X` | Run only category X (skip all others) | `Category=Button` |
| `SkipCategories=X,Y,Z` | Skip specific categories | `SkipCategories=Shell,CollectionView` |

### Platform-Specific Filter Passing

| Platform | XHarness Argument | How App Reads It |
|----------|-------------------|------------------|
| **iOS/MacCatalyst** | `--set-env=TestFilter=...` | `NSProcessInfo.ProcessInfo.Environment["TestFilter"]` |
| **Android** | `--arg TestFilter=...` | `MauiTestInstrumentation.Current.Arguments.GetString("TestFilter")` |
| **Windows** | `--filter "Category=..."` | Native vstest filter |

**Important**: iOS uses `--set-env` (environment variable), while Android uses `--arg` (instrumentation argument). These are NOT interchangeable.

### Example XHarness Commands with Filters

```bash
# iOS - uses --set-env
xharness apple test --target ios-simulator-64_18.5 --device UDID --set-env=TestFilter=Category=Button ...

# Android - uses --arg
xharness android test --package-name com.microsoft.maui.controls.devicetests --arg TestFilter=Category=Button ...
```

## Configuration Details

The `eng/helix_xharness.proj` configuration includes:
Expand Down
299 changes: 299 additions & 0 deletions .github/skills/run-device-tests/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,299 @@
---
name: run-device-tests
description: "Build and run .NET MAUI device tests locally with category filtering. Supports iOS, MacCatalyst, Android on macOS; Android, Windows on Windows. Use TestFilter to run specific test categories."
metadata:
author: dotnet-maui
version: "2.1"
compatibility: Requires xharness CLI (for iOS/MacCatalyst/Android), Xcode (for Apple platforms), Android SDK (for Android), and .NET SDK with platform workloads.
---

# Run Device Tests Skill

Build and run .NET MAUI device tests locally on iOS simulators, MacCatalyst, Android emulators, or Windows.

## Platform Support

| Host OS | Supported Platforms |
|---------|---------------------|
| macOS | ios, maccatalyst, android |
| Windows | android, windows |

## Tools Required

This skill uses `bash` together with `pwsh` (PowerShell 7+) to run the PowerShell scripts. Requires:
- `xharness` - Global dotnet tool for running tests on iOS/MacCatalyst/Android
- `dotnet` - .NET SDK with platform workloads installed
- **iOS/MacCatalyst**: Xcode with iOS simulators
- **Android**: Android SDK with emulator
- **Windows**: Windows SDK

## Dependencies

This skill uses shared infrastructure scripts:
- `.github/scripts/shared/Start-Emulator.ps1` - Detects and boots iOS simulators / Android emulators
- `.github/scripts/shared/shared-utils.ps1` - Common utility functions

These are automatically loaded by the Run-DeviceTests.ps1 script.

## When to Use

- User wants to run device tests locally
- User wants to verify iOS/MacCatalyst/Android/Windows compatibility
- User wants to test on a specific iOS version (e.g., iOS 26)
- User asks "run device tests for Controls/Core/Essentials/Graphics"
- User asks "test on iOS simulator" or "test on Android emulator"
- User asks "run device tests on MacCatalyst"
- User wants to run only specific test categories (e.g., "run Button tests")

## Available Test Projects

| Project | Path |
|---------|------|
| Controls | `src/Controls/tests/DeviceTests/Controls.DeviceTests.csproj` |
| Core | `src/Core/tests/DeviceTests/Core.DeviceTests.csproj` |
| Essentials | `src/Essentials/test/DeviceTests/Essentials.DeviceTests.csproj` |
| Graphics | `src/Graphics/tests/DeviceTests/Graphics.DeviceTests.csproj` |
| BlazorWebView | `src/BlazorWebView/tests/DeviceTests/MauiBlazorWebView.DeviceTests.csproj` |

## Scripts

All scripts are in `.github/skills/run-device-tests/scripts/`

### Run Device Tests (Full Workflow)

```bash
# Run Controls device tests on iOS simulator (default on macOS)
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform ios

# Run Core device tests on MacCatalyst
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Core -Platform maccatalyst

# Run Controls device tests on Android emulator
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform android

# Run Controls device tests on Windows (default on Windows)
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform windows

# Run on specific iOS version
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Core -Platform ios -iOSVersion 26

# Run with test filter
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform ios -TestFilter "Category=Button"

# Run other test projects
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Essentials -Platform android
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Graphics -Platform maccatalyst
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project BlazorWebView -Platform ios
```

### Build Only (No Test Run)

```bash
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform ios -BuildOnly
```

### List Available Simulators/Emulators

```bash
# iOS simulators
xcrun simctl list devices available

# Android emulators
emulator -list-avds
```

## Workflow

1. **Run tests**: `scripts/Run-DeviceTests.ps1 -Project <name> -Platform <platform>` automatically detects/boots device, builds, and runs tests
2. **Check results**: Look at the console output or `artifacts/log/` directory for detailed test results

## Output

- Build artifacts: `artifacts/bin/<Project>.DeviceTests/<Configuration>/<tfm>/<rid>/`
- Test logs: `artifacts/log/`
- Test results summary is printed to console

## Prerequisites

- `xharness` global tool: `dotnet tool install --global Microsoft.DotNet.XHarness.CLI`
- .NET SDK with platform workloads
- **iOS/MacCatalyst**: Xcode with simulators installed
- **Android**: Android SDK with emulator configured
- **Windows**: Windows SDK
- For iOS 26: macOS Tahoe (26) with Xcode 26

## Examples

```bash
# Quick test run for Controls on iOS (default on macOS)
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform ios

# Test on MacCatalyst
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform maccatalyst

# Test on Android emulator (works on both macOS and Windows)
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform android

# Test on Windows (default on Windows)
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform windows

# Test on iOS 26 specifically
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform ios -iOSVersion 26

# Run only Button category tests on iOS
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform ios -TestFilter "Category=Button"

# Build for Android without running tests
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Core -Platform android -BuildOnly
```

## Notes

- The script automatically detects and boots an iOS simulator / Android emulator using the shared Start-Emulator.ps1 infrastructure
- Default iOS simulator is iPhone Xs with iOS 18.5 (same as UI tests)
- Default Android emulator priority: API 30 Nexus > API 30 > Nexus > First available
- MacCatalyst runs directly on the Mac (no simulator needed)
- Windows tests run directly on the local machine
- Simulator/emulator selection and boot logic is handled by `.github/scripts/shared/Start-Emulator.ps1`
- xharness manages test execution and reporting for iOS/MacCatalyst/Android
- Windows uses vstest for test execution

## Test Filtering

The `-TestFilter` parameter allows running specific test categories instead of all tests. This is useful for quick iteration during development.

### Filter Syntax

| Format | Description | Example |
|--------|-------------|---------|
| `Category=X` | Run only category X | `Category=Button` |
| `SkipCategories=X,Y,Z` | Skip categories X, Y, Z | `SkipCategories=Shell,CollectionView` |

### Examples

```bash
# Run only Button tests on iOS
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform ios -TestFilter "Category=Button"

# Run only Button tests on Android
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform android -TestFilter "Category=Button"

# Skip heavy test categories
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform ios -TestFilter "SkipCategories=Shell,CollectionView,HybridWebView"
```

### How Test Filtering Works

Test filtering is implemented in `src/Core/tests/DeviceTests.Shared/DeviceTestSharedHelpers.cs`:

| Platform | How Filter is Passed | How Filter is Read |
|----------|---------------------|-------------------|
| **iOS/MacCatalyst** | `--set-env=TestFilter=...` | `NSProcessInfo.ProcessInfo.Environment["TestFilter"]` |
| **Android** | `--arg TestFilter=...` | `MauiTestInstrumentation.Current.Arguments.GetString("TestFilter")` |
| **Windows** | `--filter "Category=..."` | Native vstest filter |

### Available Test Categories

Common categories in Controls.DeviceTests:
- `Button`, `Label`, `Entry`, `Editor` - Individual control tests
- `CollectionView`, `ListView`, `CarouselView` - Collection controls (heavy)
- `Shell`, `Navigation`, `TabbedPage` - Navigation tests (heavy)
- `Layout`, `FlexLayout` - Layout tests
- `Memory` - Memory leak tests
- `Accessibility` - Accessibility tests
- `Gesture` - Gesture recognition tests

To see all categories, check `src/Controls/tests/DeviceTests/TestCategory.cs`.

## Troubleshooting

### Xcode Version Mismatch

If you see errors about Xcode version mismatch (e.g., "Xcode 26.2 installed but 26.0 required"):

```bash
# Use -SkipXcodeVersionCheck to bypass validation
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform ios -SkipXcodeVersionCheck
```

### MacCatalyst App Bundle Not Found

MacCatalyst apps use display names (e.g., "Controls Tests.app") not assembly names. The script handles this automatically by searching for `.app` bundles in the output directory.

### Android Package Name Issues

Android package names must be lowercase (e.g., `com.microsoft.maui.controls.devicetests`). The script has a mapping of project names to correct package names.

### Test Filter Not Working

1. **Ensure full rebuild**: The test filter code must be compiled into the app. Delete artifacts and rebuild:
```bash
rm -rf artifacts/bin/<Project>.DeviceTests
pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform android -TestFilter "Category=Button"
```

2. **Check the console output**: Look for `TestFilter: Category=Button` in the test output to confirm filter was applied.

3. **Verify category exists**: Ensure the category name matches exactly (case-sensitive).

## XHarness Device Detection

The script automatically handles XHarness device targeting for iOS and Android:

### iOS
1. **Simulator Boot**: Start-Emulator.ps1 detects and boots appropriate iOS simulator
2. **UDID Extraction**: Script extracts simulator UDID from Start-Emulator.ps1 output
3. **iOS Version Detection**: Script queries `xcrun simctl` to get iOS version from booted simulator
4. **XHarness Targeting**: Passes both `--target ios-simulator-64_VERSION` and `--device UDID` to xharness for explicit targeting

### MacCatalyst
- Runs directly on the Mac, no device targeting needed
- XHarness uses `--target maccatalyst`

### Android
1. **Emulator Boot**: Start-Emulator.ps1 detects running device or boots emulator
2. **Device ID Extraction**: Script gets device ID (e.g., `emulator-5554`)
3. **XHarness Targeting**: Passes `--device-id` to xharness android command

### Windows
- No device/emulator needed
- Uses vstest (`dotnet test`) for test execution

**Why both --target and --device for iOS?**
- XHarness requires `--target ios-simulator-64` (or `ios-simulator-64_VERSION`) to specify platform type
- Adding `--device UDID` explicitly tells xharness which simulator to use
- This combination ensures reliable device selection even on ARM64 Macs where automatic detection can fail

**Example xharness invocations:**
```bash
# iOS with test filter
dotnet xharness apple test \
--app path/to/app \
--target ios-simulator-64_18.5 \
--device 56AE278D-60F7-4892-9DE0-6341357CA068 \
-o artifacts/log \
--timeout 01:00:00 \
-v \
--set-env=TestFilter=Category=Button

# MacCatalyst with test filter
dotnet xharness apple test \
--app path/to/app \
--target maccatalyst \
-o artifacts/log \
--timeout 01:00:00 \
-v \
--set-env=TestFilter=Category=Button

# Android with test filter
dotnet xharness android test \
--app path/to/app.apk \
--package-name com.microsoft.maui.controls.devicetests \
--device-id emulator-5554 \
-o artifacts/log \
--timeout 01:00:00 \
-v \
--arg TestFilter=Category=Button
```

This ensures tests run on the correct device with proper version targeting and filtering.
Loading
Loading