Unify acceptance test data sources into [TestMatrix] and [CompatibilityMatrix] - #16174
Merged
Jakub Jareš (nohwnd) merged 5 commits intoJun 30, 2026
Conversation
…tyMatrix] The acceptance tests had nine data-source attributes that overlapped and read inconsistently - Runner vs VSTestConsole, exclusion bools you had to mentally invert. Collapse them into two that read positively: [TestMatrix(console, testHost, ...)] for the framework matrix and [CompatibilityMatrix(scenario)] for the version-compat matrix. You pin an axis instead of excluding one. Migrated all in-repo call sites (48 files, a literal name swap). The old attributes stay for now so the open PRs that still use them keep building; they come out in a follow-up once those have merged. Checked the new attributes against the old ones for every call shape - same frameworks, /InIsolation, VSIX rows and order all match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR consolidates acceptance/integration test data-source attributes into two unified attributes—[TestMatrix] for local runner/testhost framework combinations and [CompatibilityMatrix] for shipped-version compatibility sweeps—then migrates in-repo call sites to use the new attributes while keeping legacy attributes in place for compatibility with in-flight PRs.
Changes:
- Added
TestMatrixAttributeto generate the local framework matrix (console × testhost, plus in-isolation/in-process/VSIX variants where applicable). - Added
CompatibilityMatrixAttributeto replace multiple*CompatibilityDataSourceattributes with a single scenario-driven attribute. - Migrated acceptance/integration tests to the new attributes (literal attribute swaps; intended no behavioral change).
Reviewed changes
Copilot reviewed 50 out of 50 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| test/Microsoft.TestPlatform.TestUtilities/TestMatrixAttribute.cs | Introduces unified local runner/testhost matrix data source ([TestMatrix]). |
| test/Microsoft.TestPlatform.TestUtilities/CompatibilityMatrixAttribute.cs | Introduces unified shipped-version compatibility matrix data source ([CompatibilityMatrix]). |
| test/Microsoft.TestPlatform.Library.IntegrationTests/TranslationLayerTests/TargetFrameworkTestHostDemultiplexer.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Library.IntegrationTests/TranslationLayerTests/SerializeTestRunTests.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Library.IntegrationTests/TranslationLayerTests/RunTestsWithFilterTests.cs | Replaces compatibility and per-TFM data sources with [CompatibilityMatrix]/[TestMatrix]. |
| test/Microsoft.TestPlatform.Library.IntegrationTests/TranslationLayerTests/RunTestsWithDifferentConfigurationTests.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Library.IntegrationTests/TranslationLayerTests/RunTests.cs | Replaces compatibility and per-TFM data sources with [CompatibilityMatrix]/[TestMatrix]. |
| test/Microsoft.TestPlatform.Library.IntegrationTests/TranslationLayerTests/RunSelectedTests.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Library.IntegrationTests/TranslationLayerTests/LiveUnitTestingTests.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Library.IntegrationTests/TranslationLayerTests/DiscoverTests.cs | Replaces compatibility and per-TFM data sources with [CompatibilityMatrix]/[TestMatrix]. |
| test/Microsoft.TestPlatform.Library.IntegrationTests/TranslationLayerTests/DifferentTestFrameworkSimpleTests.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Library.IntegrationTests/TranslationLayerTests/DataCollectorAttachmentProcessor.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Library.IntegrationTests/TranslationLayerTests/CustomTestHostLauncherTests.cs | Replaces compatibility data sources with [CompatibilityMatrix]. |
| test/Microsoft.TestPlatform.Library.IntegrationTests/TranslationLayerTests/CodeCoverageTests.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Library.IntegrationTests/FilterSourceIntegrationTests.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Library.IntegrationTests/AppDomainTests.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/VideoRecorderTests.cs | Replaces legacy matrix attribute with [TestMatrix] (incl. VSIX). |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/TestPlatformNugetPackageTests.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/TestCaseFilterTests.cs | Replaces per-TFM data sources with [TestMatrix] usage (incl. in-process). |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/TelemetryTests.cs | Replaces per-TFM data sources with [TestMatrix] usage (incl. in-process). |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/SerializerSelectionTests.cs | Replaces runner-specific data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/SerializationCompatibilityTests.cs | Replaces multiple compatibility data sources with [CompatibilityMatrix]. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/SelfContainedAppTests.cs | Replaces per-TFM data sources with [TestMatrix] usage (net console only). |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/RunsettingsTests.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/ResultsDirectoryTests.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/RecursiveResourcesLookupTests.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/ProcessesInteractionTests.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/PortableNugetPackageTests.cs | Replaces per-TFM data sources with [TestMatrix] usage (incl. in-process). |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/PlatformTests.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/MultitargetingTestHostTests.cs | Replaces runner-specific attributes with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/MissingTestSdkTests.cs | Replaces runner-specific data sources with [TestMatrix] usage (net console only). |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/LoggerTests.cs | Replaces per-TFM data sources with [TestMatrix] usage (incl. in-process). |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/ListExtensionsTests.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/FrameworkTests.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/FilePatternParserTests.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/ExecutionThreadApartmentStateTests.cs | Replaces per-TFM data sources with [TestMatrix] usage (incl. in-process). |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/ExecutionTests.cs | Replaces compatibility and per-TFM data sources with [CompatibilityMatrix]/[TestMatrix]. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/EventLogCollectorTests.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DotnetTestTests.cs | Replaces per-TFM data sources with [TestMatrix] usage (net console only). |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DotnetTestMSBuildOutputTests.cs | Replaces per-TFM data sources with [TestMatrix] usage (net console only). |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DiscoveryTests.cs | Replaces per-TFM data sources with [TestMatrix] usage (incl. in-process/VSIX). |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DisableAppdomainTests.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DifferentTestFrameworkSimpleTests.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DebugAssertTests.cs | Replaces per-TFM data sources with [TestMatrix] usage (net console only). |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DataCollectorTests.Coverlet.cs | Replaces runner-specific data sources with [TestMatrix] usage (net console only). |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DataCollectionTests.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/CreateNoNewWindowTests.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/CodeCoverageTests.cs | Replaces per-TFM data sources with [TestMatrix] usage (net console only). |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/BlameDataCollectorTests.cs | Replaces runner-specific and per-TFM data sources with [TestMatrix] usage. |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/ArgumentProcessorTests.cs | Replaces per-TFM data sources with [TestMatrix] usage. |
…rosoft#16128) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previously the VSIX row was nested under the NetFx console/host axes, so combinations like console: Net or testHost: Net silently dropped it. Emit it whenever vsix: true (Windows-only), independent of the axes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolve conflicts from microsoft#16113 (Remove Windows-Review from cross-platform tests). Took origin/main as the semantic baseline for testhost selection and Windows-Review placement, then translated the old data-source attributes to the new [TestMatrix]/[CompatibilityMatrix] vocabulary so both PRs' intents are preserved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
VSTestConsole and TestHost were structurally identical ({ Both, NetFx, Net }).
Collapse them into one Target enum and global-using-static it in the two
integration test projects, so call sites read
[TestMatrix(console: NetFx, testHost: Net)]
The console:/testHost: parameter names carry the axis the type used to.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Amaury Levé (Evangelink)
approved these changes
Jun 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Our acceptance tests grew nine data-source attributes that overlap and read inconsistently —
RunnervsVSTestConsole, and exclusion bools you have to mentally invert (useDesktopRunner: falseto mean "net console only"). This folds them into two that read positively:[TestMatrix(console, testHost, inIsolation, inProcess, vsix)]— the framework matrix (replacesNetCore/NetFullTargetFrameworkDataSource,NetCoreRunner,NetFrameworkRunner). You pin an axis instead of excluding one, e.g.[TestMatrix(testHost: TestHost.Net)]or[TestMatrix(console: VSTestConsole.Net)].[CompatibilityMatrix(CompatScenario.{VSTestConsole,TestHost,Wrapper,Adapter})]— the version-compat matrix (replaces the five*CompatibilityDataSourceattributes).All in-repo call sites are migrated — a literal name swap, no behavior change. I kept the old attributes in place so the PRs currently in flight that still use them keep building; they come out in a follow-up once those have merged. No
[Obsolete], because we build warnings-as-errors.I checked the new attributes against the old ones for every call shape — frameworks,
/InIsolation, VSIX rows and order all match.