feat(engine): raise thread pool minimum when blocking tests starve the pool

docs/docs/execution/parallelism.md

@@ -18,6 +18,10 @@ With no attributes, every test is eligible to run concurrently. The .NET thread
 :::note Async tests and the thread pool
 TUnit uses the standard .NET thread pool and, like most .NET applications, is susceptible to thread pool starvation. Blocking calls or heavy CPU-bound `Task.Run` work can delay async continuations in other tests and cause timed waits to expire.
 
+By default, the pool adds threads beyond its minimum only about once or twice per second, so a burst of blocking tests can delay timers and `[Timeout]` handling in unrelated tests. To reduce this, TUnit watches the thread pool while tests run. If queued work waits for a thread while most CPUs are idle, TUnit raises the pool's minimum worker thread count to the maximum number of parallel tests plus one per processor. If the pool is still starved once it has grown that far (for example because keyed `[NotInParallel]` tests run alongside the parallel limit), TUnit doubles the minimum, up to 64 threads per processor. The raised minimum lasts for the rest of the run. Suites that do not block never trigger this, so they keep the default thread count. TUnit never lowers a minimum that you configured yourself. To turn this off, set `context.Settings.Execution.ThreadPoolTuningEnabled = false` in a `[Before(HookType.TestDiscovery)]` hook (see [Programmatic Configuration](../reference/programmatic-configuration.md)), or set the `TUNIT_DISABLE_THREADPOOL_TUNING=1` environment variable.
+
+When a test times out, the error message includes a thread pool snapshot. If the pool looked saturated, the message says the timeout may come from blocking calls in other tests.
+
 Use proper async throughout your tests: `await` asynchronous APIs instead of blocking with `.Wait()`, `.Result`, or `.GetAwaiter().GetResult()`. Await asynchronous I/O directly rather than wrapping it in `Task.Run`; `Task.Run` still uses the same thread pool.
 
 For heavy CPU-bound tests, use a shared [`ParallelLimiter<T>`](#parallellimitert--limiting-concurrent-test-count) to reduce concurrency for those tests. The limiter caps concurrent tests that share it, not the number of tasks each test creates.

docs/docs/reference/environment-variables.md

@@ -132,6 +132,18 @@ export TUNIT_MAX_PARALLEL_TESTS=0    # Unlimited parallelism
 
 **Note:** Command-line arguments take precedence over environment variables.
 
+### TUNIT_DISABLE_THREADPOOL_TUNING
+
+By default, TUnit watches the thread pool while tests run. If queued work waits for a thread while most CPUs are idle (a sign that tests are blocking), TUnit raises the pool's minimum worker thread count to the maximum number of parallel tests plus one per processor, and doubles it while starvation continues, up to 64 per processor. This stops blocking tests from delaying timers and timeouts in other tests while the pool slowly adds threads. TUnit never lowers a minimum that you configured yourself. Set this variable to `1`, `true` or `yes` to turn the behavior off.
+
+```bash
+export TUNIT_DISABLE_THREADPOOL_TUNING=1
+```
+
+**Programmatic equivalent:** `context.Settings.Execution.ThreadPoolTuningEnabled = false` (see [Programmatic Configuration](./programmatic-configuration.md))
+
+**Note:** The environment variable takes precedence over the programmatic setting.
+
 ### TUNIT_EXECUTION_MODE
 
 Controls whether tests run in source-generation (AOT-optimized) mode or reflection mode.
@@ -305,6 +317,7 @@ When the same setting is configured in multiple places, TUnit follows this prior
 | `TUNIT_DISABLE_ARTIFACT_UPLOAD` | - | Keeps the HTML report file but skips the GitHub Actions artifact upload (`context.Settings.Reporting.ArtifactUploadEnabled = false`) |
 | `JUNIT_XML_OUTPUT_PATH` | - | JUnit output path |
 | `TUNIT_MAX_PARALLEL_TESTS` | `--maximum-parallel-tests` | Max parallel tests |
+| `TUNIT_DISABLE_THREADPOOL_TUNING` | - | Disables the thread pool starvation monitor (`1` to disable) |
 | `TUNIT_EXECUTION_MODE` | `--reflection` | Selects source-generation (`sourcegeneration`/`aot`) or `reflection` execution mode |
 | `TUNIT_DISCOVERY_DIAGNOSTICS` | - | Enables discovery-time diagnostics (`1` to enable) |
 | `TUNIT_DIAGNOSTIC_CAST` | - | Enables `CastHelper` conversion diagnostics (`true` to enable) |

docs/docs/reference/programmatic-configuration.md

@@ -76,6 +76,9 @@ On Linux and macOS with .NET 8 or later, SIGTERM (sent by `docker stop`, Kuberne
 | Property | Type | Default | Description |
 |---|---|---|---|
 | `FailFast` | `bool` | `false` | Cancels the remaining test run after the first test failure. |
+| `ThreadPoolTuningEnabled` | `bool` | `true` | Raises the thread pool's minimum worker thread count when blocking tests starve the pool. See [Controlling Parallelism](../execution/parallelism.md). |
+
+The `TUNIT_DISABLE_THREADPOOL_TUNING` environment variable takes precedence over `ThreadPoolTuningEnabled`.
 
 ### `context.Settings.Reporting`
 

src/TUnit.Core/Settings/ExecutionSettings.cs

@@ -12,4 +12,13 @@ internal ExecutionSettings() { }
     /// Precedence: <c>--fail-fast</c> → TUnitSettings → built-in default.
     /// </summary>
     public bool FailFast { get; set; }
+
+    /// <summary>
+    /// Whether TUnit raises the thread pool's minimum worker thread count when blocking tests
+    /// starve the pool. Starvation delays timers and <c>[Timeout]</c> handling in unrelated tests.
+    /// The minimum is raised only after starvation is detected (and doubled, up to 64 threads per
+    /// processor, while it continues), so suites that never block are unaffected. Default: <c>true</c>.
+    /// Precedence: <c>TUNIT_DISABLE_THREADPOOL_TUNING</c> → TUnitSettings → built-in default.
+    /// </summary>
+    public bool ThreadPoolTuningEnabled { get; set; } = true;
 }

src/TUnit.Engine/Configuration/EnvironmentConstants.cs

@@ -29,6 +29,9 @@ internal static class EnvironmentConstants
     // TUnit-specific: Execution
     public const string ExecutionMode = "TUNIT_EXECUTION_MODE";
     public const string MaxParallelTests = "TUNIT_MAX_PARALLEL_TESTS";
+    // Opts out of the thread pool starvation monitor (raises min worker threads when tests block).
+    // Takes precedence over TUnitSettings.Execution.ThreadPoolTuningEnabled.
+    public const string DisableThreadPoolTuning = "TUNIT_DISABLE_THREADPOOL_TUNING";
 
     // TUnit-specific: Display and diagnostics
     public const string DisableLogo = "TUNIT_DISABLE_LOGO";

src/TUnit.Engine/Configuration/EnvironmentFlags.cs

@@ -0,0 +1,16 @@
+namespace TUnit.Engine.Configuration;
+
+internal static class EnvironmentFlags
+{
+    /// <summary>
+    /// The truthy vocabulary shared by the TUNIT_DISABLE_* switches: <c>1</c>, <c>true</c> or <c>yes</c>.
+    /// </summary>
+    public static bool IsTruthy(string? value)
+        => value is not null &&
+           (value.Equals("true", StringComparison.OrdinalIgnoreCase) ||
+            value.Equals("1", StringComparison.Ordinal) ||
+            value.Equals("yes", StringComparison.OrdinalIgnoreCase));
+
+    public static bool IsSet(string variableName)
+        => IsTruthy(Environment.GetEnvironmentVariable(variableName));
+}

src/TUnit.Engine/Helpers/TimeoutDiagnostics.cs

@@ -41,6 +41,11 @@ public static string BuildTimeoutDiagnosticsMessage(
         var sb = new StringBuilder(baseMessage);
 
         AppendTaskStatus(sb, executionTask, executionException);
+#if NET
+        ThreadPool.GetMaxThreads(out var maxWorkerThreads, out _);
+        ThreadPool.GetAvailableThreads(out var availableWorkerThreads, out _);
+        AppendThreadPoolDiagnostics(sb, ThreadPool.ThreadCount, maxWorkerThreads - availableWorkerThreads, ThreadPool.PendingWorkItemCount);
+#endif
         AppendStackTraceDiagnostics(sb);
 
         return sb.ToString();
@@ -89,6 +94,49 @@ private static void AppendTaskException(StringBuilder sb, Exception exception)
         sb.Append(exception.Message);
     }
 
+#if NET
+    /// <summary>
+    /// Appends a thread pool snapshot. A backlog of queued work means continuations and timer
+    /// callbacks were waiting for a thread, so the timeout may be caused by other tests blocking.
+    /// </summary>
+    internal static void AppendThreadPoolDiagnostics(StringBuilder sb, int threadCount, int busyWorkerThreads, long pendingWorkItems)
+    {
+        ThreadPool.GetMinThreads(out var minWorkerThreads, out _);
+        ThreadPool.GetMaxThreads(out var maxWorkerThreads, out _);
+
+        sb.AppendLine();
+        sb.AppendLine();
+        sb.Append("--- Thread Pool ---");
+        sb.AppendLine();
+        sb.Append("Threads: ");
+        sb.Append(threadCount);
+        sb.Append(", busy: ");
+        sb.Append(busyWorkerThreads);
+        sb.Append(", pending work items: ");
+        sb.Append(pendingWorkItems);
+        sb.Append(", min worker threads: ");
+        sb.Append(minWorkerThreads);
+        sb.Append(", max worker threads: ");
+        sb.Append(maxWorkerThreads);
+
+        if (IsThreadPoolSaturated(threadCount, busyWorkerThreads, pendingWorkItems))
+        {
+            sb.AppendLine();
+            sb.Append("The thread pool appears saturated. This timeout may be caused by blocking calls ");
+            sb.Append("(.Result, .Wait(), Thread.Sleep, synchronous I/O) in other concurrently running tests ");
+            sb.Append("rather than by this test. Use async APIs throughout, or limit concurrency for heavy tests ");
+            sb.Append("with [ParallelLimiter<T>] or --maximum-parallel-tests.");
+        }
+    }
+
+    /// <summary>
+    /// Work is waiting and every pool thread is busy. Even one queued timer callback counts: it
+    /// cannot run until a thread frees up.
+    /// </summary>
+    internal static bool IsThreadPoolSaturated(int threadCount, int busyWorkerThreads, long pendingWorkItems)
+        => pendingWorkItems > 0 && busyWorkerThreads >= threadCount;
+#endif
+
     private static void AppendStackTraceDiagnostics(StringBuilder sb)
     {
         string stackTrace;

src/TUnit.Engine/Reporters/Html/HtmlReporter.cs

@@ -448,23 +448,16 @@ private static void TryDeleteReportFile(Action deleteFile)
         }
     }
 
-    // The truthy vocabulary shared by the TUNIT_DISABLE_* switches.
-    private static bool IsTruthyEnv(string? value)
-        => value is not null &&
-           (value.Equals("true", StringComparison.OrdinalIgnoreCase) ||
-            value.Equals("1", StringComparison.Ordinal) ||
-            value.Equals("yes", StringComparison.OrdinalIgnoreCase));
-
     internal static bool IsHtmlReportEnabled()
-        => !IsTruthyEnv(Environment.GetEnvironmentVariable(EnvironmentConstants.DisableHtmlReporter))
+        => !EnvironmentFlags.IsSet(EnvironmentConstants.DisableHtmlReporter)
            && TUnitSettings.Default.Reporting.HtmlReportEnabled;
 
     internal static bool IsJsonReportEnabled()
-        => !IsTruthyEnv(Environment.GetEnvironmentVariable(EnvironmentConstants.DisableJsonReport))
+        => !EnvironmentFlags.IsSet(EnvironmentConstants.DisableJsonReport)
            && TUnitSettings.Default.Reporting.JsonReportEnabled;
 
     internal static bool IsArtifactUploadEnabled()
-        => !IsTruthyEnv(Environment.GetEnvironmentVariable(EnvironmentConstants.DisableArtifactUpload))
+        => !EnvironmentFlags.IsSet(EnvironmentConstants.DisableArtifactUpload)
            && TUnitSettings.Default.Reporting.ArtifactUploadEnabled;
 
     internal bool IsHtmlReportEnabledForRun()

src/TUnit.Engine/Scheduling/TestScheduler.cs

@@ -103,6 +103,10 @@ public async Task<bool> ScheduleAndExecuteAsync(
             return true;
         }
 
+        // Covers every phase that runs user code (static property initializers, tests,
+        // session hooks) so blocking tests do not delay timers in other tests.
+        using var starvationMonitor = ThreadPoolStarvationMonitor.Start(_maxParallelism.Value, _logger);
+
         // Initialize static properties before tests run
         await _staticPropertyHandler.InitializeStaticPropertiesAsync(cancellationToken).ConfigureAwait(false);