Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,14 +61,14 @@ Source generation shifts work from run time to build time: you pay a little up f
<!-- benchmarks:start -->
| Scenario | TUnit (AOT) | TUnit | xUnit v3 | NUnit | MSTest |
|----------|---|---|---|---|---|
| Data-driven tests | 17.74 ms | 308.59 ms | 534.14 ms | 547.52 ms | 552.18 ms |
| Async-heavy tests | 120.7 ms | 406.5 ms | 671.9 ms | 657.7 ms | 750.7 ms |
| Matrix combinations | 115.4 ms | 375.1 ms | 1,443.5 ms | 1,420.2 ms | 1,530.7 ms |
| Large suites (scale) | 15.92 ms | 272.51 ms | 500.67 ms | 502.50 ms | 494.11 ms |
| Massive parallelism | 217.8 ms | 469.3 ms | 2,926.8 ms | 1,073.9 ms | 2,981.5 ms |
| Setup/teardown lifecycle | 66.11 ms | 327.59 ms | 1,024.32 ms | 996.24 ms | 1,066.16 ms |

<sub>Mean wall-clock time to run the same test suite. TUnit (AOT) 1.64.0 · TUnit 1.64.0 · xUnit v3 3.2.2 · NUnit 4.6.1 · MSTest 4.3.3. .NET SDK 10.0.302, .NET 10.0.10 (10.0.10, 10.0.1026.32716), X64 RyuJIT x86-64-v4. Updated 2026-08-09 — regenerated weekly by the [Speed Comparison workflow](https://github.com/thomhurst/TUnit/actions/workflows/speed-comparison.yml). Full results and methodology: [tunit.dev/docs/benchmarks](https://tunit.dev/docs/benchmarks/).</sub>
| Data-driven tests | 13.53 ms | 318.76 ms | 680.19 ms | 540.84 ms | 577.30 ms |
| Async-heavy tests | 118.5 ms | 437.3 ms | 906.6 ms | 653.1 ms | 724.2 ms |
| Matrix combinations | 117.8 ms | 436.2 ms | 1,677.9 ms | 1,471.3 ms | 1,547.8 ms |
| Large suites (scale) | 19.74 ms | 349.50 ms | 752.17 ms | 581.67 ms | 577.69 ms |
| Massive parallelism | 218.0 ms | 566.0 ms | 3,101.4 ms | 1,151.5 ms | 3,028.4 ms |
| Setup/teardown lifecycle | 66.91 ms | 355.61 ms | 1,292.08 ms | 1,045.37 ms | 1,104.79 ms |

<sub>Mean wall-clock time to run the same test suite. TUnit (AOT) 1.65.0 · TUnit 1.65.0 · xUnit v3 4.0.0 · NUnit 4.6.1 · MSTest 4.3.3. .NET SDK 10.0.400, .NET 10.0.11 (10.0.11, 10.0.1126.37416), X64 RyuJIT x86-64-v4. Updated 2026-08-16 — regenerated weekly by the [Speed Comparison workflow](https://github.com/thomhurst/TUnit/actions/workflows/speed-comparison.yml). Full results and methodology: [tunit.dev/docs/benchmarks](https://tunit.dev/docs/benchmarks/).</sub>
<!-- benchmarks:end -->

## Getting Started
Expand Down
20 changes: 10 additions & 10 deletions docs/docs/benchmarks/engine/AsyncTests.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,20 +9,20 @@ sidebar_position: 3
> Realistic async/await patterns with I/O simulation

:::info Last Updated
This benchmark was automatically generated on **2026-08-09** from the latest CI run.
This benchmark was automatically generated on **2026-08-16** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.302
**Environment:** Ubuntu Latest • .NET SDK 10.0.400
:::

## 📊 Results

| Framework | Version | Mean | Median | StdDev |
|-----------|---------|------|--------|--------|
| **TUnit** | 1.64.0 | 406.5 ms | 401.4 ms | 20.76 ms |
| NUnit | 4.6.1 | 657.7 ms | 649.5 ms | 38.73 ms |
| MSTest | 4.3.3 | 750.7 ms | 745.6 ms | 38.09 ms |
| xUnit3 | 3.2.2 | 671.9 ms | 670.0 ms | 22.31 ms |
| **TUnit (AOT)** | 1.64.0 | 120.7 ms | 120.7 ms | 1.57 ms |
| **TUnit** | 1.65.0 | 437.3 ms | 431.8 ms | 39.79 ms |
| NUnit | 4.6.1 | 653.1 ms | 655.6 ms | 60.47 ms |
| MSTest | 4.3.3 | 724.2 ms | 726.6 ms | 42.51 ms |
| xUnit3 | 4.0.0 | 906.6 ms | 906.0 ms | 45.02 ms |
| **TUnit (AOT)** | 1.65.0 | 118.5 ms | 118.3 ms | 1.24 ms |

## 📈 Visual Comparison

Expand Down Expand Up @@ -60,8 +60,8 @@ This benchmark was automatically generated on **2026-08-09** from the latest CI
xychart-beta
title "AsyncTests Performance Comparison"
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
y-axis "Time (ms)" 0 --> 901
bar [406.5, 657.7, 750.7, 671.9, 120.7]
y-axis "Time (ms)" 0 --> 1088
bar [437.3, 653.1, 724.2, 906.6, 118.5]
```

## 🎯 Key Insights
Expand All @@ -74,4 +74,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
:::

*Last generated: 2026-08-09T00:23:17.026Z*
*Last generated: 2026-08-16T00:18:52.418Z*
18 changes: 9 additions & 9 deletions docs/docs/benchmarks/engine/BuildTime.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ sidebar_position: 9
> Compilation time from a clean build across frameworks — how long it takes to build an identical test project.

:::info Last Updated
This benchmark was automatically generated on **2026-08-09** from the latest CI run.
This benchmark was automatically generated on **2026-08-16** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.302
**Environment:** Ubuntu Latest • .NET SDK 10.0.400
:::

## 📊 Results
Expand All @@ -20,10 +20,10 @@ Compilation time comparison across frameworks:

| Framework | Version | Mean | Median | StdDev |
|-----------|---------|------|--------|--------|
| **TUnit** | 1.64.0 | 940.2 ms | 938.3 ms | 27.85 ms |
| Build_NUnit | 4.6.1 | 925.3 ms | 924.0 ms | 17.10 ms |
| Build_MSTest | 4.3.3 | 1,033.0 ms | 1,024.9 ms | 49.06 ms |
| Build_xUnit3 | 3.2.2 | 930.5 ms | 927.8 ms | 16.14 ms |
| **TUnit** | 1.65.0 | 1,022.9 ms | 1,015.7 ms | 54.92 ms |
| Build_NUnit | 4.6.1 | 914.6 ms | 915.7 ms | 6.63 ms |
| Build_MSTest | 4.3.3 | 1,115.7 ms | 1,115.6 ms | 52.23 ms |
| Build_xUnit3 | 4.0.0 | 929.3 ms | 926.6 ms | 12.60 ms |

## 📈 Visual Comparison

Expand Down Expand Up @@ -61,8 +61,8 @@ Compilation time comparison across frameworks:
xychart-beta
title "Build Time Comparison"
x-axis ["Build_TUnit", "Build_NUnit", "Build_MSTest", "Build_xUnit3"]
y-axis "Time (ms)" 0 --> 1240
bar [940.2, 925.3, 1033, 930.5]
y-axis "Time (ms)" 0 --> 1339
bar [1022.9, 914.6, 1115.7, 929.3]
```

---
Expand All @@ -71,4 +71,4 @@ xychart-beta
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
:::

*Last generated: 2026-08-09T00:23:17.061Z*
*Last generated: 2026-08-16T00:18:52.420Z*
20 changes: 10 additions & 10 deletions docs/docs/benchmarks/engine/DataDrivenTests.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,20 +9,20 @@ sidebar_position: 4
> Parameterized tests with multiple data sources

:::info Last Updated
This benchmark was automatically generated on **2026-08-09** from the latest CI run.
This benchmark was automatically generated on **2026-08-16** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.302
**Environment:** Ubuntu Latest • .NET SDK 10.0.400
:::

## 📊 Results

| Framework | Version | Mean | Median | StdDev |
|-----------|---------|------|--------|--------|
| **TUnit** | 1.64.0 | 308.59 ms | 307.89 ms | 5.350 ms |
| NUnit | 4.6.1 | 547.52 ms | 541.49 ms | 31.636 ms |
| MSTest | 4.3.3 | 552.18 ms | 551.76 ms | 41.191 ms |
| xUnit3 | 3.2.2 | 534.14 ms | 529.06 ms | 9.859 ms |
| **TUnit (AOT)** | 1.64.0 | 17.74 ms | 17.58 ms | 0.933 ms |
| **TUnit** | 1.65.0 | 318.76 ms | 312.55 ms | 15.439 ms |
| NUnit | 4.6.1 | 540.84 ms | 535.88 ms | 30.877 ms |
| MSTest | 4.3.3 | 577.30 ms | 574.52 ms | 30.305 ms |
| xUnit3 | 4.0.0 | 680.19 ms | 676.84 ms | 51.572 ms |
| **TUnit (AOT)** | 1.65.0 | 13.53 ms | 13.47 ms | 0.228 ms |

## 📈 Visual Comparison

Expand Down Expand Up @@ -60,8 +60,8 @@ This benchmark was automatically generated on **2026-08-09** from the latest CI
xychart-beta
title "DataDrivenTests Performance Comparison"
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
y-axis "Time (ms)" 0 --> 663
bar [308.59, 547.52, 552.18, 534.14, 17.74]
y-axis "Time (ms)" 0 --> 817
bar [318.76, 540.84, 577.3, 680.19, 13.53]
```

## 🎯 Key Insights
Expand All @@ -74,4 +74,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
:::

*Last generated: 2026-08-09T00:23:17.026Z*
*Last generated: 2026-08-16T00:18:52.418Z*
20 changes: 10 additions & 10 deletions docs/docs/benchmarks/engine/MassiveParallelTests.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,20 +9,20 @@ sidebar_position: 5
> Parallel execution stress tests

:::info Last Updated
This benchmark was automatically generated on **2026-08-09** from the latest CI run.
This benchmark was automatically generated on **2026-08-16** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.302
**Environment:** Ubuntu Latest • .NET SDK 10.0.400
:::

## 📊 Results

| Framework | Version | Mean | Median | StdDev |
|-----------|---------|------|--------|--------|
| **TUnit** | 1.64.0 | 469.3 ms | 469.4 ms | 3.00 ms |
| NUnit | 4.6.1 | 1,073.9 ms | 1,067.0 ms | 12.86 ms |
| MSTest | 4.3.3 | 2,981.5 ms | 2,993.4 ms | 30.52 ms |
| xUnit3 | 3.2.2 | 2,926.8 ms | 2,928.2 ms | 12.70 ms |
| **TUnit (AOT)** | 1.64.0 | 217.8 ms | 217.7 ms | 0.39 ms |
| **TUnit** | 1.65.0 | 566.0 ms | 565.5 ms | 41.65 ms |
| NUnit | 4.6.1 | 1,151.5 ms | 1,136.4 ms | 40.23 ms |
| MSTest | 4.3.3 | 3,028.4 ms | 3,010.5 ms | 33.46 ms |
| xUnit3 | 4.0.0 | 3,101.4 ms | 3,095.4 ms | 27.30 ms |
| **TUnit (AOT)** | 1.65.0 | 218.0 ms | 217.2 ms | 1.54 ms |

## 📈 Visual Comparison

Expand Down Expand Up @@ -60,8 +60,8 @@ This benchmark was automatically generated on **2026-08-09** from the latest CI
xychart-beta
title "MassiveParallelTests Performance Comparison"
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
y-axis "Time (ms)" 0 --> 3578
bar [469.3, 1073.9, 2981.5, 2926.8, 217.8]
y-axis "Time (ms)" 0 --> 3722
bar [566, 1151.5, 3028.4, 3101.4, 218]
```

## 🎯 Key Insights
Expand All @@ -74,4 +74,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
:::

*Last generated: 2026-08-09T00:23:17.027Z*
*Last generated: 2026-08-16T00:18:52.418Z*
20 changes: 10 additions & 10 deletions docs/docs/benchmarks/engine/MatrixTests.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,20 +9,20 @@ sidebar_position: 6
> Combinatorial test generation and execution

:::info Last Updated
This benchmark was automatically generated on **2026-08-09** from the latest CI run.
This benchmark was automatically generated on **2026-08-16** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.302
**Environment:** Ubuntu Latest • .NET SDK 10.0.400
:::

## 📊 Results

| Framework | Version | Mean | Median | StdDev |
|-----------|---------|------|--------|--------|
| **TUnit** | 1.64.0 | 375.1 ms | 375.9 ms | 5.58 ms |
| NUnit | 4.6.1 | 1,420.2 ms | 1,421.2 ms | 7.71 ms |
| MSTest | 4.3.3 | 1,530.7 ms | 1,525.7 ms | 22.65 ms |
| xUnit3 | 3.2.2 | 1,443.5 ms | 1,440.6 ms | 6.54 ms |
| **TUnit (AOT)** | 1.64.0 | 115.4 ms | 115.3 ms | 0.63 ms |
| **TUnit** | 1.65.0 | 436.2 ms | 430.7 ms | 28.73 ms |
| NUnit | 4.6.1 | 1,471.3 ms | 1,461.4 ms | 35.30 ms |
| MSTest | 4.3.3 | 1,547.8 ms | 1,538.3 ms | 24.02 ms |
| xUnit3 | 4.0.0 | 1,677.9 ms | 1,673.5 ms | 17.25 ms |
| **TUnit (AOT)** | 1.65.0 | 117.8 ms | 117.7 ms | 0.34 ms |

## 📈 Visual Comparison

Expand Down Expand Up @@ -60,8 +60,8 @@ This benchmark was automatically generated on **2026-08-09** from the latest CI
xychart-beta
title "MatrixTests Performance Comparison"
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
y-axis "Time (ms)" 0 --> 1837
bar [375.1, 1420.2, 1530.7, 1443.5, 115.4]
y-axis "Time (ms)" 0 --> 2014
bar [436.2, 1471.3, 1547.8, 1677.9, 117.8]
```

## 🎯 Key Insights
Expand All @@ -74,4 +74,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
:::

*Last generated: 2026-08-09T00:23:17.027Z*
*Last generated: 2026-08-16T00:18:52.419Z*
20 changes: 10 additions & 10 deletions docs/docs/benchmarks/engine/ScaleTests.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,20 +9,20 @@ sidebar_position: 7
> Large test suites (150+ tests) measuring scalability

:::info Last Updated
This benchmark was automatically generated on **2026-08-09** from the latest CI run.
This benchmark was automatically generated on **2026-08-16** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.302
**Environment:** Ubuntu Latest • .NET SDK 10.0.400
:::

## 📊 Results

| Framework | Version | Mean | Median | StdDev |
|-----------|---------|------|--------|--------|
| **TUnit** | 1.64.0 | 272.51 ms | 271.73 ms | 3.031 ms |
| NUnit | 4.6.1 | 502.50 ms | 495.10 ms | 21.289 ms |
| MSTest | 4.3.3 | 494.11 ms | 495.49 ms | 4.829 ms |
| xUnit3 | 3.2.2 | 500.67 ms | 496.60 ms | 13.537 ms |
| **TUnit (AOT)** | 1.64.0 | 15.92 ms | 15.92 ms | 0.588 ms |
| **TUnit** | 1.65.0 | 349.50 ms | 333.60 ms | 41.972 ms |
| NUnit | 4.6.1 | 581.67 ms | 574.97 ms | 30.611 ms |
| MSTest | 4.3.3 | 577.69 ms | 579.55 ms | 30.378 ms |
| xUnit3 | 4.0.0 | 752.17 ms | 752.19 ms | 62.610 ms |
| **TUnit (AOT)** | 1.65.0 | 19.74 ms | 19.67 ms | 0.613 ms |

## 📈 Visual Comparison

Expand Down Expand Up @@ -60,8 +60,8 @@ This benchmark was automatically generated on **2026-08-09** from the latest CI
xychart-beta
title "ScaleTests Performance Comparison"
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
y-axis "Time (ms)" 0 --> 603
bar [272.51, 502.5, 494.11, 500.67, 15.92]
y-axis "Time (ms)" 0 --> 903
bar [349.5, 581.67, 577.69, 752.17, 19.74]
```

## 🎯 Key Insights
Expand All @@ -74,4 +74,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
:::

*Last generated: 2026-08-09T00:23:17.060Z*
*Last generated: 2026-08-16T00:18:52.419Z*
20 changes: 10 additions & 10 deletions docs/docs/benchmarks/engine/SetupTeardownTests.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,20 +9,20 @@ sidebar_position: 8
> Expensive test fixtures with setup/teardown overhead

:::info Last Updated
This benchmark was automatically generated on **2026-08-09** from the latest CI run.
This benchmark was automatically generated on **2026-08-16** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.302
**Environment:** Ubuntu Latest • .NET SDK 10.0.400
:::

## 📊 Results

| Framework | Version | Mean | Median | StdDev |
|-----------|---------|------|--------|--------|
| **TUnit** | 1.64.0 | 327.59 ms | 326.78 ms | 2.732 ms |
| NUnit | 4.6.1 | 996.24 ms | 995.63 ms | 7.547 ms |
| MSTest | 4.3.3 | 1,066.16 ms | 1,060.72 ms | 17.292 ms |
| xUnit3 | 3.2.2 | 1,024.32 ms | 1,023.42 ms | 9.380 ms |
| **TUnit (AOT)** | 1.64.0 | 66.11 ms | 66.06 ms | 0.728 ms |
| **TUnit** | 1.65.0 | 355.61 ms | 356.54 ms | 4.842 ms |
| NUnit | 4.6.1 | 1,045.37 ms | 1,047.53 ms | 14.229 ms |
| MSTest | 4.3.3 | 1,104.79 ms | 1,107.44 ms | 8.732 ms |
| xUnit3 | 4.0.0 | 1,292.08 ms | 1,281.97 ms | 41.267 ms |
| **TUnit (AOT)** | 1.65.0 | 66.91 ms | 66.72 ms | 0.630 ms |

## 📈 Visual Comparison

Expand Down Expand Up @@ -60,8 +60,8 @@ This benchmark was automatically generated on **2026-08-09** from the latest CI
xychart-beta
title "SetupTeardownTests Performance Comparison"
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
y-axis "Time (ms)" 0 --> 1280
bar [327.59, 996.24, 1066.16, 1024.32, 66.11]
y-axis "Time (ms)" 0 --> 1551
bar [355.61, 1045.37, 1104.79, 1292.08, 66.91]
```

## 🎯 Key Insights
Expand All @@ -74,4 +74,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
:::

*Last generated: 2026-08-09T00:23:17.061Z*
*Last generated: 2026-08-16T00:18:52.419Z*
14 changes: 7 additions & 7 deletions docs/docs/benchmarks/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@ sidebar_position: 1
# Performance Benchmarks

:::info Last Updated
These benchmarks were automatically generated on **2026-08-09** from the latest CI run.
These benchmarks were automatically generated on **2026-08-16** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.302
**Environment:** Ubuntu Latest • .NET SDK 10.0.400
:::

## 🚀 Runtime Benchmarks
Expand Down Expand Up @@ -37,8 +37,8 @@ These benchmarks compare TUnit against the most popular .NET testing frameworks:

| Framework | Version Tested |
|-----------|----------------|
| **TUnit** | 1.64.0 |
| **xUnit v3** | 3.2.2 |
| **TUnit** | 1.65.0 |
| **xUnit v3** | 4.0.0 |
| **NUnit** | 4.6.1 |
| **MSTest** | 4.3.3 |

Expand All @@ -56,8 +56,8 @@ The benchmarks measure real-world testing patterns:
### Environment

- **OS**: Ubuntu Latest (GitHub Actions)
- **Runtime**: .NET 10.0.10 (10.0.10, 10.0.1026.32716), X64 RyuJIT x86-64-v4
- **SDK**: .NET SDK 10.0.302
- **Runtime**: .NET 10.0.11 (10.0.11, 10.0.1126.37416), X64 RyuJIT x86-64-v4
- **SDK**: .NET SDK 10.0.400
- **Hardware**: GitHub Actions Standard Runner (Ubuntu)
- **Tool**: BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)

Expand All @@ -80,4 +80,4 @@ These benchmarks run automatically daily via [GitHub Actions](https://github.com
Each benchmark runs multiple iterations with statistical analysis to ensure accuracy. Results may vary based on hardware and test characteristics.
:::

*Last generated: 2026-08-09T00:23:17.061Z*
*Last generated: 2026-08-16T00:18:52.420Z*
Loading
Loading