feat: add [PairwiseDataSource] for all-pairs test case generation

docs/docs/writing-tests/data-driven-overview.md

@@ -11,6 +11,7 @@ TUnit offers several ways to provide data to your tests. Use this guide to pick
 | Shared object with lifecycle | `[ClassDataSource<T>]` | [Class Data Source](class-data-source.md) |
 | Reusable data rows | `[TestDataRow<T>]` | [Test Data Row](test-data-row.md) |
 | All parameter combinations | `[MatrixDataSource]` | [Matrix Tests](matrix-tests.md) |
+| Every pair of parameter values (fewer cases) | `[PairwiseDataSource]` | [Pairwise Tests](pairwise-tests.md) |
 | Multiple sources on one method | Combined attributes | [Combined Data Sources](combined-data-source.md) |
 | Hierarchical injection | Nested properties | [Nested Data Sources](nested-data-sources.md) |
 | Custom generic attributes | `[GenerateGenericTest(typeof(...))]` | [Generic Attributes](generic-attributes.md) |
@@ -78,4 +79,5 @@ public async Task Multiply(
 - **`[MethodDataSource]`** is best for computed or complex data.
 - **`[ClassDataSource<T>]`** manages object lifecycles (initialization, disposal, sharing across tests).
 - **`[MatrixDataSource]`** generates the Cartesian product of all `[Matrix]` parameter values.
+- **`[PairwiseDataSource]`** uses the same `[Matrix]` parameters but only covers every pair of values, which is far fewer test cases when there are many parameters.
 - Attributes can be combined on a single method — see [Combined Data Sources](combined-data-source.md).

docs/docs/writing-tests/matrix-tests.md

@@ -6,6 +6,7 @@ The Matrix data source is a way to specify different arguments per parameter, an
 As your number of arguments and/or parameters increase, the number of test cases will grow exponentially.  
 For example, 3 parameters with 10 values each will generate 1,000 test cases (10 × 10 × 10).  
 Use with caution to avoid very large test suites.
+If you don't need every combination, [Pairwise Tests](pairwise-tests.md) use the same `[Matrix]` parameter attributes but only generate enough test cases to cover every pair of values.
 :::
 
 For our arguments, we'll add a `[Matrix]` attribute. Instead of this being added to the test method, it's added to the parameters themselves.

docs/docs/writing-tests/pairwise-tests.md

@@ -0,0 +1,174 @@
+# Pairwise Tests
+
+[Matrix tests](matrix-tests.md) run every combination of parameter values, so the number of test cases grows exponentially with the number of parameters.
+Pairwise testing (also known as *all-pairs* testing) is a well-known way to keep that under control: instead of every combination, it generates a much smaller set of test cases in which **every pair of values, across every pair of parameters, appears at least once**.
+
+Most bugs are triggered by a single value or by the interaction of two parameters, so pairwise testing catches a large share of them with far fewer test cases.
+
+To use it, swap `[MatrixDataSource]` for `[PairwiseDataSource]`. The parameters are declared exactly as for matrix tests:
+
+```csharp
+using TUnit.Assertions;
+using TUnit.Assertions.Extensions;
+using TUnit.Core;
+
+namespace MyTestProject;
+
+public class MyTestClass
+{
+    [Test]
+    [PairwiseDataSource]
+    public async Task MyTest(
+        bool largeArrayPool,
+        bool excessSpan,
+        [Matrix(0, 99, 100, 101)] int sizeHint,
+        [Matrix(1, 2, 3, 100)] int stepCount)
+    {
+        var total = sizeHint + stepCount;
+
+        await Assert.That(total).IsPositive();
+    }
+}
+```
+
+`[MatrixDataSource]` would generate 2 × 2 × 4 × 4 = 64 test cases for this test. `[PairwiseDataSource]` generates 16, and every combination of values for any two of the parameters (for example `excessSpan: true` with `sizeHint: 101`) is still tested.
+
+The savings grow quickly with more parameters:
+
+| Parameter value counts | `[MatrixDataSource]` | `[PairwiseDataSource]` |
+|---|---:|---:|
+| 2 × 2 × 4 × 4 | 64 | 16 |
+| 2 × 2 × 3 × 3 × 2 | 72 | 10 |
+| 3 × 3 × 3 × 3 | 81 | 9 |
+| 3 × 4 × 5 × 6 | 360 | 30 |
+| 10 × 10 × 10 | 1,000 | 106 |
+| 2 (×10 parameters) | 1,024 | 9 |
+
+:::info
+With two parameters, pairwise and matrix generate the same test cases, because every combination *is* a pair.
+Pairwise starts to pay off from three parameters.
+:::
+
+:::warning
+Pairwise testing does not test every combination. If a bug only shows up for a specific combination of three or more parameter values, a pairwise test may not hit it.
+Use `[MatrixDataSource]` when you need exhaustive coverage.
+:::
+
+## Parameter values
+
+`[PairwiseDataSource]` reads parameter values the same way as `[MatrixDataSource]`, so everything described in [Matrix Tests](matrix-tests.md) works here too:
+
+- `[Matrix(...)]` and `[Matrix<T>(...)]` with explicit values, including `Excluding = [...]`
+- `[MatrixRange<T>(...)]` for numeric ranges
+- `[MatrixMethod<T>(...)]` for values returned by a method
+- `bool` and `enum` parameters (and their nullable forms) expand to all of their values automatically. For nullable types, `null` is added after the other values.
+
+```csharp
+using TUnit.Assertions;
+using TUnit.Assertions.Extensions;
+using TUnit.Core;
+
+namespace MyTestProject;
+
+public enum Color
+{
+    Red,
+    Green,
+    Blue
+}
+
+public class MyTestClass
+{
+    [Test]
+    [PairwiseDataSource]
+    public async Task MyTest(
+        Color color,
+        bool? flag,
+        [MatrixRange<int>(1, 5)] int number,
+        [Matrix("x", "y")] string text,
+        [MatrixMethod<MyTestClass>(nameof(Sizes))] int size)
+    {
+        await Assert.That(number).IsBetween(1, 5);
+    }
+
+    public static IEnumerable<int> Sizes()
+    {
+        yield return 1;
+        yield return 10;
+        yield return 100;
+    }
+}
+```
+
+This generates 17 test cases instead of 270.
+
+## Exclusions
+
+`[MatrixExclusion(...)]` works with `[PairwiseDataSource]` as well. Exclusions are applied *while* the test cases are generated, not by filtering afterwards, so every pair of values that can still appear in an allowed test case remains covered.
+
+```csharp
+using TUnit.Assertions;
+using TUnit.Assertions.Extensions;
+using TUnit.Core;
+
+namespace MyTestProject;
+
+public class MyTestClass
+{
+    [Test]
+    [PairwiseDataSource]
+    [MatrixExclusion(1, 1, 1)]
+    [MatrixExclusion(2, 2, 2)]
+    [MatrixExclusion(3, 3, 3)]
+    public async Task MyTest(
+        [Matrix(1, 2, 3)] int value1,
+        [Matrix(1, 2, 3)] int value2,
+        [Matrix(1, 2, 3)] int value3)
+    {
+        var allEqual = value1 == value2 && value2 == value3;
+
+        await Assert.That(allEqual).IsFalse();
+    }
+}
+```
+
+## Determinism and seeds
+
+The generated test cases are deterministic: the same parameter values always produce the same test cases, so test IDs are stable between runs and between source-generated and reflection mode.
+
+The algorithm uses a seeded pseudo-random number generator to pick test cases. You can change the seed with the `Seed` property to get a different, equally valid, set of test cases:
+
+```csharp
+using TUnit.Assertions;
+using TUnit.Assertions.Extensions;
+using TUnit.Core;
+
+namespace MyTestProject;
+
+public class MyTestClass
+{
+    [Test]
+    [PairwiseDataSource(Seed = 42)]
+    public async Task MyTest(
+        [Matrix(1, 2, 3)] int a,
+        [Matrix(1, 2, 3, 4)] int b,
+        [Matrix(1, 2, 3, 4, 5)] int c)
+    {
+        await Assert.That(a + b + c).IsPositive();
+    }
+}
+```
+
+:::note
+Changing the parameter values (or their order) can change which test cases are generated, not just add or remove the cases for the changed value.
+:::
+
+## Limitations
+
+- Like `[MatrixDataSource]`, `[PairwiseDataSource]` only supports test method parameters. It can't be used for class constructor parameters.
+- Pairwise coverage only guarantees that every *pair* of values is tested. It does not try to cover combinations of three or more values.
+
+## Algorithm
+
+The pairwise algorithm is based on Bob Jenkins' [jenny](https://burtleburtle.net/bob/math/jenny.html) tool, by way of NUnit's `PairwiseStrategy` and [Xunit.Combinatorial](https://github.com/AArnott/Xunit.Combinatorial)'s `[PairwiseData]`.
+It uses the same default seed as Xunit.Combinatorial, so tests migrated from `[PairwiseData]` generate the same number of test cases.

docs/sidebars.ts

@@ -31,6 +31,7 @@ const sidebars: SidebarsConfig = {
             'writing-tests/test-data-row',
             'writing-tests/defer-enumeration',
             'writing-tests/matrix-tests',
+            'writing-tests/pairwise-tests',
             'writing-tests/combined-data-source',
             'writing-tests/nested-data-sources',
             'writing-tests/generic-attributes',

src/TUnit.Analyzers/Extensions/AttributeExtensions.cs

@@ -97,6 +97,22 @@ public static bool IsMatrixDataSourceAttribute(this AttributeData attributeData,
             TUnitSymbols.For(compilation).MatrixDataSourceAttribute);
     }
 
+    public static bool IsPairwiseDataSourceAttribute(this AttributeData attributeData, Compilation compilation)
+    {
+        return SymbolEqualityComparer.Default.Equals(attributeData.AttributeClass,
+            TUnitSymbols.For(compilation).PairwiseDataSourceAttribute);
+    }
+
+    /// <summary>
+    /// Returns true for data sources that generate test cases from the [Matrix] values on parameters
+    /// ([MatrixDataSource] and [PairwiseDataSource]).
+    /// </summary>
+    public static bool IsMatrixValuesDataSourceAttribute(this AttributeData attributeData, Compilation compilation)
+    {
+        return attributeData.IsMatrixDataSourceAttribute(compilation)
+            || attributeData.IsPairwiseDataSourceAttribute(compilation);
+    }
+
     public static bool IsCombinedDataSourceAttribute(this AttributeData attributeData, Compilation compilation)
     {
         return SymbolEqualityComparer.Default.Equals(attributeData.AttributeClass,

src/TUnit.Analyzers/Helpers/TUnitSymbols.cs

@@ -38,6 +38,7 @@ private TUnitSymbols(Compilation compilation)
         BaseTestAttribute = compilation.GetTypeByMetadataName("TUnit.Core.BaseTestAttribute");
         MatrixAttribute = compilation.GetTypeByMetadataName(WellKnown.AttributeFullyQualifiedClasses.Matrix.WithoutGlobalPrefix);
         MatrixDataSourceAttribute = compilation.GetTypeByMetadataName(WellKnown.AttributeFullyQualifiedClasses.MatrixDataSourceAttribute.WithoutGlobalPrefix);
+        PairwiseDataSourceAttribute = compilation.GetTypeByMetadataName(WellKnown.AttributeFullyQualifiedClasses.PairwiseDataSourceAttribute.WithoutGlobalPrefix);
         CombinedDataSourceAttribute = compilation.GetTypeByMetadataName(WellKnown.AttributeFullyQualifiedClasses.CombinedDataSourceAttribute.WithoutGlobalPrefix);
         DataSourceAttributeInterface = compilation.GetTypeByMetadataName(WellKnown.AttributeFullyQualifiedClasses.IDataSourceAttribute.WithoutGlobalPrefix);
     }
@@ -60,6 +61,8 @@ private TUnitSymbols(Compilation compilation)
 
     public INamedTypeSymbol? MatrixDataSourceAttribute { get; }
 
+    public INamedTypeSymbol? PairwiseDataSourceAttribute { get; }
+
     public INamedTypeSymbol? CombinedDataSourceAttribute { get; }
 
     public INamedTypeSymbol? DataSourceAttributeInterface { get; }

src/TUnit.Analyzers/Helpers/WellKnown.cs

@@ -8,6 +8,7 @@ public static class AttributeFullyQualifiedClasses
         public static readonly FullyQualifiedTypeName Explicit = GetTypeName("ExplicitAttribute");
         public static readonly FullyQualifiedTypeName Matrix = GetTypeName("MatrixAttribute");
         public static readonly FullyQualifiedTypeName MatrixDataSourceAttribute = GetTypeName("MatrixDataSourceAttribute");
+        public static readonly FullyQualifiedTypeName PairwiseDataSourceAttribute = GetTypeName("PairwiseDataSourceAttribute");
         public static readonly FullyQualifiedTypeName CombinedDataSourceAttribute = GetTypeName("CombinedDataSourcesAttribute");
 
         public static readonly FullyQualifiedTypeName BeforeAttribute = GetTypeName("BeforeAttribute");

src/TUnit.Analyzers/MatrixAnalyzer.cs

@@ -66,7 +66,7 @@ private void CheckMatrixErrors(SymbolAnalysisContext context, ImmutableArray<Att
             return;
         }
 
-        if (!attributes.Any(x => x.IsMatrixDataSourceAttribute(context.Compilation)))
+        if (!attributes.Any(x => x.IsMatrixValuesDataSourceAttribute(context.Compilation)))
         {
             context.ReportDiagnostic(
                 Diagnostic.Create(Rules.MatrixDataSourceAttributeRequired,

src/TUnit.Analyzers/Resources.resx

@@ -346,13 +346,13 @@
         <value>Test methods must not be static</value>
     </data>
     <data name="TUnit0049Description" xml:space="preserve">
-        <value>[MatrixDataSourceAttribute] is required if using [Matrix] values on your parameters.</value>
+        <value>[MatrixDataSource] or [PairwiseDataSource] is required if using [Matrix] values on your parameters.</value>
     </data>
     <data name="TUnit0049MessageFormat" xml:space="preserve">
-        <value>[MatrixDataSourceAttribute] is required if using [Matrix] values on your parameters.</value>
+        <value>[MatrixDataSource] or [PairwiseDataSource] is required if using [Matrix] values on your parameters.</value>
     </data>
     <data name="TUnit0049Title" xml:space="preserve">
-        <value>[MatrixDataSourceAttribute] is required</value>
+        <value>[MatrixDataSource] or [PairwiseDataSource] is required</value>
     </data>
     <data name="TUnit0050Description" xml:space="preserve">
         <value>Too many arguments provided.</value>