Migrate device and WASM tests from XHarness to DeviceRunners dotnet test - #4004
Merged
Conversation
Contributor
📦 Try the packages from this PRWarning Do not run these scripts without first reviewing the code in this PR. Step 1 — Download the packages bash / macOS / Linux: curl -fsSL https://raw.githubusercontent.com/mono/SkiaSharp/main/scripts/get-skiasharp-pr.sh | bash -s -- 4004PowerShell / Windows: iex "& { $(irm https://raw.githubusercontent.com/mono/SkiaSharp/main/scripts/get-skiasharp-pr.ps1) } 4004"Step 2 — Add the local NuGet source dotnet nuget add source ~/.skiasharp/hives/pr-4004/packages --name skiasharp-pr-4004More options
Or download manually from Azure Pipelines — look for the Remove the source when you're done: dotnet nuget remove source skiasharp-pr-4004 |
Contributor
|
📖 Documentation Preview The documentation for this PR has been deployed and is available at: 🔗 View Staging Site This preview will be updated automatically when you push new commits to this PR. This comment is automatically updated by the documentation staging workflow. |
mattleibow
force-pushed
the
mattleibow/dev-devicerunners-dotnet-test
branch
from
May 18, 2026 09:01
7a98398 to
0ab77d9
Compare
mattleibow
force-pushed
the
mattleibow/dev-devicerunners-dotnet-test
branch
from
May 18, 2026 12:49
0ab77d9 to
bc3a0de
Compare
mattleibow
force-pushed
the
mattleibow/dev-devicerunners-dotnet-test
branch
from
May 18, 2026 14:06
bc3a0de to
8a95b8c
Compare
mattleibow
force-pushed
the
mattleibow/dev-devicerunners-dotnet-test
branch
from
May 21, 2026 18:27
61de65b to
feff303
Compare
- Update DeviceRunners packages from 0.1.0-preview.6 to 0.1.0-preview.9 - Remove XHarness packages (DeviceRunners.XHarness.Maui, DeviceRunners.XHarness.Xunit) - Add DeviceRunners.Testing.Targets for dotnet test integration - Update MauiProgram.cs to use AddCliConfiguration() + AddConsoleResultChannel() - Rewrite WASM test project as Blazor WebAssembly with DeviceRunners visual runner - Replace custom ThreadlessXunitTestRunner with DeviceRunners.VisualRunners.Blazor - Add curated WASM smoke tests (surface, canvas, path, encoding, etc.) - Add RunDeviceRunnersTest() shared helper in test-shared.cake - Update CI scripts (tests-android, tests-apple, tests-wasm) to use shared helper - Update Azure Pipelines YAML for TRX test result format - Remove xharness CLI tool from dotnet-tools.json - Delete utils/WasmTestRunner (Selenium-based runner no longer needed) - Remove chromedriver argument from WASM CI job - Add nuget.org source to nuget.config for DeviceRunners packages Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add IsBrowser property to BaseTest using OperatingSystem.IsBrowser() - Skip GRGlInterface test that crashes WASM runtime (native abort) - Add DefaultTestConfig.Wasm.cs so PathRoot is non-null in WASM - Add test-results to .gitignore Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The TrimmerRootAssembly with RootMode=all preserved code paths calling native functions not available in the WASM binary, causing 'null function' crashes during cleanup. Default Blazor WASM trimming works correctly — xUnit reflection-based discovery finds all 3853 tests without overrides. Removed WasmTestConfig as it depends on the shared test project's TestConfig which isn't needed for discovery. File-dependent tests fail gracefully with null PathRoot (466 expected failures). Results: 3853 total, 3219 passed, 466 failed, 168 skipped, clean exit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
DeviceRunners 0.1.0-preview.9 WASM CLI doesn't produce TRX files yet, and 466 tests fail expectedly due to missing filesystem content. The allowFailure parameter lets CI continue while still capturing the browser-console.log with full test results (3219/3853 pass). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Embed test content (fonts + images) as EmbeddedResources and extract them to the Emscripten VFS at startup via WasmTestConfig, matching the same pattern used by the device tests (AssetCopier). Results: 3595 passed, 0 failed, 258 skipped (19.4s) Skipped tests are genuine WASM platform limitations: - No system font manager (font family matching tests) - No emoji fonts (unicode glyph tests) - Mono GC doesn't guarantee finalizer timing (SkipOnMono) - Minor color rounding differences in bitmap copy Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add WasmEnableSIMD=true to WASM test project to use SIMD-enabled native binary (st,simd variant) instead of scalar-only (st) binary - Remove blanket SkipOnPlatform(IsBrowser) from CopyToSucceeds and CopyWithAlphaToSucceeds — with SIMD most color types now pass - Add AssertSimilarColor helper to SKTest base class for ±1 tolerance assertions on color values (Gray8 rounding differs by 1 between x86 SSE and WASM SIMD128 implementations) - Use AssertSimilarColor(tolerance: 1) for Gray8 test cases instead of exact equality Results: 3647 passed, 0 failed, 206 skipped (was 3595/0/258) Net gain: +52 tests passing, -52 tests skipped Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
iOS: Boot a simulator before running tests (DeviceRunners CLI requires a booted simulator or explicit --device flag). Mac Catalyst: Pass explicit RuntimeIdentifier to work around DeviceRunners targets not finding .app in RID subdirectory. Also extends RunDeviceRunnersTest to accept MSBuild properties for platform-specific configuration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use 'dotnet apple simulator boot' which works out of the box with pre-installed simulators on CI agents. No need for manual UDID tracking or xcrun calls. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Added appledev.tools 0.8.10 to .config/dotnet-tools.json so 'dotnet tool restore' installs it on CI agents - Moved simulator boot/shutdown into the tests-ios task body instead of Setup/Teardown so Mac Catalyst doesn't try to boot a simulator - Mac Catalyst runs directly without simulator dependency Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
DeviceRunners targets handle the OutputPath and .app discovery natively. Passing RuntimeIdentifier caused build to target a specific RID subfolder where the .app wasn't being produced. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
iOS: Switch from 'dotnet apple simulator boot' to 'xcrun simctl'
which is native to macOS and reliably waits for boot completion
via 'xcrun simctl bootstatus'. The dotnet-apple tool was reporting
success but the simulator wasn't actually booted when tests ran.
Mac Catalyst: DeviceRunners searches for '$(AssemblyName).app' but
the MAUI SDK creates the bundle using ApplicationTitle ('SkiaSharp
Tests.app' with a space). Align ApplicationTitle with AssemblyName
so DeviceRunners can discover the .app bundle.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The test projects don't need strict Xcode version matching - we want to be able to build and test with any compatible Xcode version. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
DeviceRunners VisualRunners has a thread-safety issue in TestAssemblyViewModel that causes SIGSEGV during GC when tests run in parallel (see mattleibow/DeviceRunners#132). Disabling parallelization avoids the race condition entirely. Results: 3746 passed, 0 failed, 194 skipped (complete run, no crash). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Update all DeviceRunners packages from preview.9 to preview.11 - Remove xunit.runner.json workaround (threading fix in preview.10) - Remove ApplicationTitle workaround (app discovery fix in preview.10) - Add android.permission.INTERNET for TCP result channel - Add explicit Activity Name for DeviceRunners CLI app launch - Add adb reverse port forwarding in Android test setup - Make TakeSnapshot non-fatal (don't abort tests on screencapture failure) Local test results (all platforms, parallel enabled): - Mac Catalyst: 3737 passed, 0 failed - iOS: 3746 passed, 0 failed - WASM: 3683 passed, 0 failed - Android: 3730 passed, 0 failed Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mattleibow
force-pushed
the
mattleibow/dev-devicerunners-dotnet-test
branch
from
May 26, 2026 09:01
837ed29 to
84778ae
Compare
The Android IL linker is very memory-intensive. Running it concurrently with the emulator caused the CI agent to exceed 95% memory and get killed. Fix by pre-building (including IL linking) before starting the emulator, then running tests with --no-build. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mattleibow
force-pushed
the
mattleibow/dev-devicerunners-dotnet-test
branch
from
May 26, 2026 14:14
247ca7e to
30af01b
Compare
The old XHarness script used configuration='Debug' for iOS and Mac Catalyst tests. Release mode enables the trimmer which strips P/Invoke methods from SkiaApi, causing ApiTest parameterized theories to discover only 658 methods instead of the full 1250. This restores the 5707 test count on Mac Catalyst (was 3931 in Release). Verified locally: Debug gives 5707 total tests matching main exactly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove the WASM allowFailure escape hatch so dotnet test failures propagate through Cake like every other DeviceRunners target. WASM now passes normally, so failures should fail the pipeline instead of being logged as warnings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Migrates SkiaSharp’s device (Android/iOS/Mac Catalyst) and WASM test execution from XHarness/custom runners to DeviceRunners’ dotnet test integration so CI produces TRX/VSTest results and failures propagate reliably.
Changes:
- Replaces XHarness + custom WASM Selenium runner with DeviceRunners-based
dotnet testflows (shared Cake helper + updated CI publishing to TRX). - Converts
SkiaSharp.Tests.Wasminto a Blazor WebAssembly DeviceRunners test host that runs the shared SkiaSharp test suite, with WASM-specific config/content extraction and targeted skips. - Updates device test app wiring/config (MAUI runner configuration, Android activity/manifest tweaks, iOS simulator lifecycle via
dotnet apple).
Reviewed changes
Copilot reviewed 40 out of 41 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| utils/WasmTestRunner/WasmTestRunner.csproj | Removes obsolete Selenium-based WASM test runner project. |
| utils/WasmTestRunner/Program.cs | Removes obsolete Selenium-based WASM test runner implementation. |
| utils/Utils.sln | Drops the removed WasmTestRunner project from the utils solution. |
| utils/README.md | Removes WasmTestRunner documentation section. |
| tests/Tests/SkiaSharp/SKTypefaceTest.cs | Skips emoji/system-font-dependent glyph test on browser/WASM. |
| tests/Tests/SkiaSharp/SKTest.cs | Adds color comparison helper with tolerance for pixel-diff variability. |
| tests/Tests/SkiaSharp/SKPaintTest.cs | Adds browser/WASM skips for font and pixel-diff-sensitive tests. |
| tests/Tests/SkiaSharp/SKImageTest.cs | Skips a finalizer-timing-dependent test on Mono platforms (incl. browser). |
| tests/Tests/SkiaSharp/SKFontTest.cs | Adds browser/WASM skips for emoji/system-font-dependent font tests. |
| tests/Tests/SkiaSharp/SKFontStyleSetTest.cs | Adds browser/WASM skips where system font manager is unavailable. |
| tests/Tests/SkiaSharp/SKFontManagerTest.cs | Adds browser/WASM skips where system font manager is unavailable. |
| tests/Tests/SkiaSharp/SKBitmapTest.cs | Uses tolerant color assertions for Gray8 pixel comparisons. |
| tests/Tests/SkiaSharp/GRGlInterfaceTest.cs | Skips a WASM-hostile native GL entry point without WebGL canvas context. |
| tests/Tests/BaseTest.cs | Adds IsBrowser detection and treats browser as Mono for skip helper. |
| tests/SkiaSharp.Tests.Wasm/Xunit/ThreadlessXunitTestRunner.cs | Removes custom threadless xUnit runner in favor of DeviceRunners. |
| tests/SkiaSharp.Tests.Wasm/wwwroot/main.js | Removes old JS-based xUnit logging interop bootstrap. |
| tests/SkiaSharp.Tests.Wasm/wwwroot/main.css | Removes old CSS used by the custom JS xUnit runner UI. |
| tests/SkiaSharp.Tests.Wasm/wwwroot/index.html | Switches WASM host page to DeviceRunners/Blazor-based runner bootstrap. |
| tests/SkiaSharp.Tests.Wasm/WasmTests.cs | Adds WASM smoke tests for basic SkiaSharp/HarfBuzz functionality. |
| tests/SkiaSharp.Tests.Wasm/WasmTestConfig.cs | Adds WASM-specific TestConfig + embedded content extraction into VFS. |
| tests/SkiaSharp.Tests.Wasm/Tests.cs | Removes prior placeholder WASM tests that didn’t run the shared suite. |
| tests/SkiaSharp.Tests.Wasm/SkiaSharp.Tests.Wasm.csproj | Converts project to Blazor WASM DeviceRunners host + references shared tests/content. |
| tests/SkiaSharp.Tests.Wasm/runtimeconfig.template.json | Removes old custom runtime config template no longer needed. |
| tests/SkiaSharp.Tests.Wasm/Properties/launchSettings.json | Removes launch settings tied to the old run model. |
| tests/SkiaSharp.Tests.Wasm/Program.cs | Boots DeviceRunners visual runner and registers shared + WASM test assemblies. |
| tests/SkiaSharp.Tests.Devices/SkiaSharp.Tests.Devices.csproj | Updates to DeviceRunners preview.11 packages and enables dotnet-test entrypoint behavior. |
| tests/SkiaSharp.Tests.Devices/Platforms/Android/MainActivity.cs | Sets explicit Android activity name for predictable runner wiring. |
| tests/SkiaSharp.Tests.Devices/Platforms/Android/AndroidManifest.xml | Adds INTERNET permission needed for TCP-based result streaming. |
| tests/SkiaSharp.Tests.Devices/MauiProgram.cs | Replaces XHarness runner with DeviceRunners visual runner + CLI config + console channel. |
| tests/Directory.Build.props | Disables Xcode version validation for tests build context. |
| scripts/infra/tests/tests-wasm.cake | Replaces Selenium runner flow with DeviceRunners dotnet test execution. |
| scripts/infra/tests/tests-apple.cake | Replaces XHarness Apple test execution with DeviceRunners dotnet test + simulator lifecycle. |
| scripts/infra/tests/tests-android.cake | Replaces XHarness Android execution with DeviceRunners dotnet test, adds prebuild + adb reverse. |
| scripts/infra/tests/test-shared.cake | Adds shared Cake helper for DeviceRunners-based dotnet test runs producing TRX. |
| scripts/infra/shared/shared.cake | Makes screenshot capture non-fatal to avoid failing on hosts without GUI support. |
| scripts/infra/caching/repo-deps.json | Removes deleted WasmTestRunner from dependency/caching inputs. |
| scripts/azure-templates-stages-test.yml | Switches test publishing from xUnit XML to VSTest TRX; removes chromedriver arg for WASM. |
| nuget.config | Adds nuget.org source so DeviceRunners packages/tools can restore. |
| documentation/dev/updating-dotnet-version.md | Removes WasmTestRunner from the dotnet-version update checklist. |
| .gitignore | Ignores WASM test-results output directory. |
| .config/dotnet-tools.json | Removes XHarness tool and adds appledev.tools for dotnet apple usage. |
Remove the temporary nuget.org source because SkiaSharp restores from mirrored feeds, drop the stale WASM test-results ignore entry, and rely on the Android emulator host gateway instead of adb reverse for DeviceRunners TCP result streaming. Use the shared RunProcess helper for iOS simulator creation so process failures are handled consistently, and narrow the finalizer skip to Browser instead of all Mono device platforms. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Align the DeviceRunners dotnet test helper with the existing build helpers by restoring through the repo package cache and mirrored/local package sources. Also route the Android pre-build through RunDotNetBuild so it uses the same restore settings, binlog, and logger behavior as the rest of the infra. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep DeviceRunners dotnet test restores on the repo package cache without forcing RestoreSources, since nuget.config already defines the mirrored feeds and output/nugets is not needed for these test projects. Route Android pre-build through RunDotNetBuild and preserve the existing SKIP_BUILD behavior for device and WASM targets. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use the same GetNuGetSources-based RestoreSources wiring as the existing build helpers for DeviceRunners dotnet test runs. This keeps the new helper aligned with the current infrastructure; any broader cleanup of output/nugets source behavior can happen separately. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrates SkiaSharp's device tests (Android, iOS, Mac Catalyst) and WASM tests from XHarness/custom launchers to DeviceRunners
0.1.0-preview.11with the newdotnet testintegration.The WASM test project now runs the shared SkiaSharp test suite through DeviceRunners instead of the old placeholder/custom Selenium runner. Device and WASM runs now produce TRX/VSTest results, and test failures propagate through Cake so CI fails when any platform test fails.
Changes
Device tests (
SkiaSharp.Tests.Devices)DeviceRunners.Testing.Targetsfordotnet testexecution.GenerateTestingPlatformEntryPoint=falsefor the MAUI test app entry point.INTERNETpermissionMainActivityname10.0.2.2) injected by DeviceRunners for host connectivitydotnet test --no-buildto avoid CI memory pressure while the emulator is running.Debug, matching the previous XHarness path. This is required to preserve fullApiTestP/Invoke reflection coverage; Release mode hidesLibraryImportsource-generated P/Invoke stubs behind local functions and drops Mac Catalyst test discovery from 5707 to 3931 tests.dotnet apple simulatorfor iOS simulator create/boot/delete flow, with simulator creation using the shared CakeRunProcesshelper.WASM tests (
SkiaSharp.Tests.Wasm)utils/WasmTestRunnerlauncher.AddXunit(useReflection: true)for browser execution where file-system based xUnit discovery is not available.CI and scripts
RunDeviceRunnersTest()Cake helper for DeviceRunners-baseddotnet testtargets.dotnet test.dotnet test.microsoft.dotnet.xharness.cli, old WASM runner files, launch settings/runtimeconfig files, and stale utility references.Validation
main:main)3889 total / 3683 passed / 0 failed.System.Private.CoreLibAOT module did not match the assembly). Removing the arm64bin/objoutput and rebuilding restored a passing5707 total / 5514 passed / 0 failedrun.