feat(engine): raise thread pool minimum when blocking tests starve the pool

[thomhurst]

Problem

Suites with timing, cancellation, or deadline-based tests time out or flake when many tests run in parallel. The cause is thread pool starvation, not yielding: an await that yields frees its thread.

• The scheduler runs up to ProcessorCount * 4 tests concurrently, and test bodies run inline on pool threads.
• The pool's minimum worker threads defaults to ProcessorCount. Beyond that, it adds threads at only about one or two per second.
• When tests block (Thread.Sleep, .Result, sync I/O), timer callbacks (Task.Delay, CancelAfter, TUnit's own [Timeout]) also wait for a thread. Timing-sensitive tests in unrelated classes then overshoot their deadlines.

Fix

ThreadPoolStarvationMonitor runs while the scheduler executes tests.

• A dedicated thread samples every 50ms. It queues a probe work item on the global queue, which is where timer callbacks wait.
• If the probe waits about a full interval (40ms or more) while process CPU utilization is under 50%, threads are blocked rather than busy.
• The monitor then raises the minimum worker threads to the parallel test limit plus ProcessorCount, in one step.
• If starvation continues after the pool has actually grown to that minimum, the monitor doubles it, up to 64 threads per processor and never above ThreadPool.GetMaxThreads. Keyed [NotInParallel] tests and dynamic tests run alongside the limited parallel phase, so more tests than the limit can block at once. Escalation waits for the pool to catch up, so raises never stack.
• The minimum observed just before the first raise is restored when the run ends, unless user code changed it.
• A user-configured minimum is never lowered.
• Opt out with context.Settings.Execution.ThreadPoolTuningEnabled = false in a [Before(HookType.TestDiscovery)] hook, or with TUNIT_DISABLE_THREADPOOL_TUNING=1. The environment variable takes precedence.
• The monitor only runs on Windows, Linux, macOS and FreeBSD. On browser, WASI, mobile and netstandard2.0 it is a no-op. The sampling thread catches all exceptions, so a failure cannot crash the test process.

Timeout diagnostics now include a thread pool snapshot: threads, busy workers, pending work items, and min/max workers. When work is queued and every pool thread is busy, the message says the timeout may come from blocking calls in other tests.

Why reactive instead of a static SetMinThreads

• A static raise at run start costs about 15% wall time on benchmarks/TUnit.PerformanceBenchmarks (12k tests, 2.63s → 3.02s), because extra pool threads spin and contend. Even min=24 on 20 cores was measurably slower.
• Pending-work-item counts are not a usable trigger. On that benchmark the engine keeps a small queue at about 25% CPU without any test blocking.
• The first raise must be a single step to the target. When threads are already in cooperative blocking (Task.Wait/.Result), the runtime hands SetMinThreads to its blocking adjustment (PortableThreadPool.SetMinThreads / Blocking.cs). Incremental raises then left the pool stuck at about 22 threads for seconds in about 40% of runs on a mixed Sleep + GetResult workload. A single raise was reliable in every run after the change. Escalation therefore only raises again once ThreadPool.ThreadCount has reached the previous minimum.

Measurements (20 cores, A/B on the same binary via the opt-out env var)

Scenario	Off	On
PerformanceBenchmarks wall time (12 runs each, alternating)	2935ms	2926ms
Blocking repro: mean p99 Task.Delay(50) overshoot (10 runs each)	73ms	48ms
Blocking repro: worst overshoot	232ms	75ms
Blocking repro: wall time	baseline	~0.8s faster

The blocking repro ran 240 blocking tests alongside 240 timer probes.

Escalation, measured with 60 blocking tests on distinct [NotInParallel] keys plus 40 timer probes, at --maximum-parallel-tests 4 (first target 24 threads), 4 runs each:

	Off	On
p99 Task.Delay(50) overshoot	133–275ms	29–58ms
Peak pool threads	40–47	64–68 (escalated past 24)
Wall time	4.2–5.7s	4.1–4.2s

The PerformanceBenchmarks A/B was re-run after escalation was added and was still neutral (3686ms off vs 3462ms on, on a noisier machine).

Testing

• New ThreadPoolStarvationMonitorTests (raise and restore, escalation gating, user-changed minimums, clamping to the pool maximum, opt-outs, env flag parsing) and the existing TimeoutHelperTests pass on net10.0 and net8.0.
• tests/TUnit.PublicAPI snapshots are updated for the one new public property, ExecutionSettings.ThreadPoolTuningEnabled.
• TUnitSettingsTests cover the new default and check that the monitor does not start when the setting is off.
• Engine integration tests TimeoutTests1, HookTimeoutTests, and DefaultTimeoutClassificationTests pass.
• The full TUnit.Engine.Tests run fails FSharp.Test. The same 7 F# project tests fail locally with the monitor on and off, so this change does not cause it.
• AOT publish was not verified locally. The monitor uses only Thread, ThreadPool, Environment.CpuUsage, and Process, with no reflection.

Metadata collection is unchanged, so source-gen and reflection modes are unaffected.

🤖 Generated with Claude Code

Summary by CodeRabbit

Summary by CodeRabbit

• New Features
  • TUnit can raise the thread pool’s minimum worker count when queued work and low CPU usage indicate starvation during a test run. Tuning is enabled by default and can be disabled through a setting or environment variable.
• Bug Fixes
  • Timeout messages now include thread-pool status and may indicate possible saturation.
• Documentation
  • Added guidance on thread-pool tuning, timeout diagnostics, and configuration options.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-11T20:39:45.473895Z	3eb4412	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: aa71ee49-e202-44ab-9e8a-713224e23171

---

📥 Commits

Reviewing files that changed from the base of the PR and between 9a54757 and 3eb4412.

---

📒 Files selected for processing (1)

• tests/TUnit.UnitTests/ThreadPoolStarvationMonitorTests.cs

---

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 5 remain after this review.

---

---

📝 Walkthrough

Walkthrough

TUnit now monitors for thread-pool starvation during test execution. Under configured probe-delay and CPU-utilization conditions, the monitor can raise the worker-thread minimum. A setting and an environment variable control monitoring. Timeout diagnostics include thread-pool state and a conditional saturation hint.

Changes

Thread-pool tuning and diagnostics

Layer / File(s)	Summary
Configure and start the monitor src/TUnit.Core/Settings/ExecutionSettings.cs, src/TUnit.Engine/Configuration/*, src/TUnit.Engine/Scheduling/TestScheduler.cs, src/TUnit.Engine/Reporters/Html/HtmlReporter.cs, tests/TUnit.UnitTests/TUnitSettingsTests.cs, tests/TUnit.PublicAPI/Tests.Core_Library_Has_No_API_Changes.*.verified.txt, docs/docs/reference/environment-variables.md, docs/docs/reference/programmatic-configuration.md	ExecutionSettings.ThreadPoolTuningEnabled defaults to true. TUNIT_DISABLE_THREADPOOL_TUNING disables monitoring and takes precedence over the setting. Shared environment-flag handling is also used for HTML report and upload disable flags. The scheduler starts the monitor before static-property initialization.
Detect starvation and restore the minimum src/TUnit.Engine/Scheduling/ThreadPoolStarvationMonitor.cs, tests/TUnit.UnitTests/ThreadPoolStarvationMonitorTests.cs	The monitor raises the worker-thread minimum when a queued probe waits at least 40 ms and CPU utilization is below 50%. It uses an initial target based on configured parallelism plus processor-count headroom, then can double the minimum up to 64 workers per processor. Disposal restores the prior minimum only if the current minimum still matches the monitor’s last raise. Tests cover startup, tuning, restoration, limits, starvation, and saturation.
Report thread-pool state in timeout diagnostics src/TUnit.Engine/Helpers/TimeoutDiagnostics.cs, docs/docs/execution/parallelism.md, docs/docs/reference/environment-variables.md, docs/docs/reference/programmatic-configuration.md	Timeout messages report thread counts, worker-thread limits, and pending work items. They include a saturation hint when pending work is positive and busy worker threads are at least the current thread count. The documentation describes monitoring controls and diagnostics.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant TestScheduler
  participant ThreadPoolStarvationMonitor
  participant ThreadPool
  participant ProcessCpuUsage
  TestScheduler->>ThreadPoolStarvationMonitor: Start with configured parallelism and logger
  ThreadPoolStarvationMonitor->>ThreadPool: Queue global-pool probe
  ThreadPoolStarvationMonitor->>ProcessCpuUsage: Sample CPU utilization
  ThreadPool-->>ThreadPoolStarvationMonitor: Return probe delay
  ThreadPoolStarvationMonitor->>ThreadPool: Raise worker-thread minimum when conditions are met
  TestScheduler->>ThreadPoolStarvationMonitor: Dispose after test execution

Loading

---

Merge Risk: 🔵 Low · up to 3eb44

Runs with a preconfigured thread-pool minimum can raise it sooner than intended. This is a bounded tuning risk that can be accepted with owner awareness or corrected before merge.

Pre-merge checks | 4 | 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 18.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 50 functions across 9 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main change: raising the thread-pool minimum when blocking tests cause starvation.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

---

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

---

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

---

_{A rabbit watched the workers queue,
While probes hopped through the thread-pool too.
When CPUs rested and waits grew,
The minimum rose by one, then two.
At timeout, pool counts came to light,
Then carrots capped the run at night.}

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Review: ThreadPoolStarvationMonitor

I reviewed the diff and found no blocking issues. I did not build the PR or run its tests. The code-review skill failed to run, so this is a manual read.

What I checked

• Design: the reactive trigger is a sound choice. A probe on the global queue, combined with low CPU utilization, targets blocked threads rather than busy ones. The PR description gives data for why a static SetMinThreads and a pending-count trigger don't work.
• Dedicated thread: a dedicated sampling thread is the right call, since a pool-based sampler would starve along with the pool.
• Raise and restore: Raise and Dispose share one lock and _stopped, so they can't race. The original minimum is restored only if the current minimum still equals the raised value, so a user-set minimum is never overwritten.
• Opt-out and guards: the env var opt-out, the browser and netstandard2.0 no-ops, and the early return when the user's minimum is already high enough are all handled.
• Lock type: Lock is already used elsewhere in the engine, so it isn't a new dependency.

Non-blocking suggestions

1. IsThreadPoolSaturated (pending >= threadCount) is a heuristic. A test timeout can fire when the queue is briefly deep for unrelated reasons, so the message may sometimes blame other tests wrongly. The wording already says "may", which is acceptable. You could also note whether the monitor raised the minimum, which would make the diagnostic more precise.
2. The monitor never samples again after one raise. If the first raise happens early and a later phase starves the pool at a higher level, nothing responds. That is a reasonable trade-off given the measurements, but it is worth a short comment or doc note.
3. The tests cover only the pure predicates and the message formatting. There is no end-to-end test that blocking tests actually trigger a raise and the restore. A small test that calls Start, blocks threads and then checks GetMinThreads would guard against regressions. It would be timing-sensitive, so keeping it out is also defensible.
4. AOT publish wasn't verified, as the PR notes. The APIs used are AOT-safe, so I'd expect no problems.

Overall this looks good to merge.

[greptile-apps]

Confidence Score: 5/5

[High impact] The latest changes appear safe to merge; no new actionable issue was found.

Summary

The PR adds a monitor that raises the thread pool minimum when blocking tests delay queued work. Timeout messages also include the pool’s current state.

• TUnit raises the worker minimum when blocked tests starve the thread pool.
• Timeout messages show when blocked workers may be delaying test callbacks.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Start test run] --> B[Sample queued probe and CPU use]
  B --> C{Probe delayed and CPUs mostly idle?}
  C -->|No| B
  C -->|Yes| D[Raise worker minimum]
  D --> E{Pool reached minimum but still starved?}
  E -->|No| B
  E -->|Yes| F[Double minimum up to ceiling]
  F --> B
  B --> G[Run ends]
  G --> H[Restore saved minimum unless user changed it]

Loading

_{Reviews (6) · Last reviewed commit: "test(engine): keep escalation tests vali..." · Reviewed by Greptile}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8f8da2a342

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

Review: ThreadPoolStarvationMonitor

I read the monitor itself; I did not run the tests or build the PR. The reactive design is sound and the PR description explains it well. The probe is queued on the global queue, where timer callbacks wait. A raise is gated on low CPU, which separates blocked threads from busy ones. It raises once, then stops sampling. It also restores the original minimum on dispose and never lowers a user-set one. The opt-outs (env var and setting) and the timeout diagnostics are useful.

Notes, none blocking:

1. Single-shot raise. After one raise the monitor stops sampling for the rest of the run. That fits the cap you chose, and the comment explains why. If SetMinThreads fails or the current minimum is already at or above the cap, Run still returns and the Raise() result is discarded. That is harmless, but it could log at debug level so the case is diagnosable.
2. Threshold edge case. _stopSignal.Wait(SampleInterval) can return slightly early or late relative to the 50ms threshold in IsStarved. A probe queued one tick ago can read as just under 50ms, which delays detection by one more tick. Use a threshold slightly below the interval (for example 0.9×), or compare against two consecutive stale ticks explicitly. Either makes the behaviour deterministic.
3. net8 allocation. Below NET9, Process.GetCurrentProcess() allocates on every 50ms sample. The cost is small and only paid until the first raise. Caching a single Process instance for the monitor's lifetime would avoid it.
4. Tests. Coverage of IsStarved and the settings is good. An integration-style test that blocks N threads and asserts the minimum was raised, then restored after Dispose, would guard the end-to-end path. It can be timing-sensitive, so a generous timeout and a skip on single-core machines would help.

LGTM overall.

[github-actions]

Review: ThreadPoolStarvationMonitor

I read ThreadPoolStarvationMonitor.cs in full. I did not read the rest of the diff closely, and I did not build or run any tests. The code-review skill failed to run, so this is a manual review.

The design holds up.

• Reactive raise. The monitor raises the minimum only after a probe work item waits a full interval while CPU is under 50%. That avoids the roughly 15% wall-time cost you measured for a static SetMinThreads.
• Dedicated thread. Sampling keeps working when every pool thread is blocked.
• Single raise. One raise to the cap fits the cooperative-blocking behaviour you describe.
• Restore on dispose. The original minimum is put back only if user code hasn't changed it in the meantime.
• Failure handling. An exception on the sampling thread is swallowed, so it can't take down the process.
• Opt-outs. The environment variable and the setting both work, and the variable takes precedence.
• Platforms. The platform gating and the netstandard2.0 no-op are handled.

Suggestions, none blocking:

1. Silent failure. Run() swallows exceptions with no logging. A _logger.LogDebug there would help diagnose cases where the monitor quietly stops.
2. No log at the raise. The only debug message is written in Dispose. Logging when Raise() succeeds would give you the timing relative to the tests that were starved, which is useful when investigating flakes.
3. Only one raise. If the first raise is not enough because the parallel limit is under-sized for blocking-heavy suites, nothing escalates further. That fits the stated design, so I'd just document it.
4. Test coverage. IsStarved and StarvedCpuUtilization are internal, which suggests they're unit-tested. An integration test that blocks threads and asserts the minimum is raised and then restored would guard the restore path.

Looks good overall.

[thomhurst]

Responding to the claude-review notes:

• Threshold jitter: fixed in 2b5515b. The probe-wait threshold is now 40ms, below the 50ms sample interval, so a wait that returns slightly early no longer delays detection by a sample.
• net8 allocation: fixed in 2b5515b. Below .NET 9, one Process instance is reused (with Refresh()) for the monitor's lifetime.
• Tests: d175009 added deterministic tests for raise to cap, restore on dispose, preserving user-set minimums, and both opt-outs. I've left out a test that really blocks threads to trigger a raise, because its timing would be flaky in CI. The end-to-end numbers in the PR description come from such a run done locally.
• Single-shot raise: intentional. The raise goes straight to the cap (parallel limit plus cores), so a second raise would have nothing left to add. The code comment explains why it must be a single step.
• Saturation hint heuristic: agreed it's a heuristic. The message says "may", which I think is the right level of confidence.
• AOT: still not verified locally, because this machine has no MSVC linker. The APIs involved use no reflection.

[github-actions]

Review: ThreadPoolStarvationMonitor

I read the diff and found no blocking problems. The design is sound.

What I like

• Reacting to measured starvation avoids the ~15% wall-time cost of a static SetMinThreads.
• The probe goes on the global queue, which is where timer callbacks wait, so it measures the latency that matters. The CPU-utilization gate separates blocked threads from busy ones.
• A dedicated thread does the sampling, and Run swallows exceptions so a failure cannot crash the process.
• Raise and Dispose share a lock and a _stopped flag. Dispose restores the minimum only if user code hasn't changed it since.
• There are opt-outs through both an environment variable and a setting, and the public API snapshots are updated.

Points to consider (non-blocking)

1. Unbounded parallelism. If _maxParallelism is effectively unlimited (for example TUNIT_MAX_PARALLEL_TESTS=0), the cap becomes short.MaxValue. A raise would then set the minimum to ~32k threads. That is likely too aggressive, and it removes the runtime's thread-injection throttling entirely. A sane ceiling would help, such as Math.Min(cap, ProcessorCount * 64). Alternatively, skip tuning when parallelism is unlimited.
2. Single raise, never re-evaluated. The monitor stops after one raise. That is fine given the comments about cooperative blocking. A one-line note in the docs that detection is once per run would set expectations.
3. Env-var parsing. IsTruthyEnv accepts 1, true and yes, but the docs only mention 1. Either document the other values or reuse an existing env-flag helper if the engine has one, so parsing stays consistent across TUNIT_* variables.
4. Process-wide CPU. CPU utilization is measured for the whole process. Background GC or other non-test work can push it over 50% and suppress the raise. This is a reasonable heuristic and fails safe, so it is probably fine. Keep it in mind if flakiness reports come in.
5. Tests. The unit tests cover Raise, Dispose and the settings opt-out. An integration-style test that starves the pool with blocking tasks and asserts the minimum was raised would also confirm the probe and CPU gate work end to end. It could be timing-sensitive, so mark it [NotInParallel] as the existing tests are.

The timeout diagnostics addition is useful and low risk.

Overall this looks good to merge, with the unbounded-parallelism cap as the one item I would address.

[thomhurst]

Responding to the latest claude-review notes, addressed in e97f2e3:

• Unbounded parallelism: 0 / "unlimited" already resolves to ProcessorCount * 4 in TestScheduler.GetMaxParallelism, so the cap is never short.MaxValue from that path. An explicitly oversized --maximum-parallel-tests value did reach a cap of thousands of threads, though, so the cap is now clamped to 64 per processor (ComputeMaxWorkerThreads, unit-tested).
• Logging: debug logs now record the raise (with before/after values), the restore, and the monitor thread stopping after an error.
• Env-var parsing: the 1/true/yes parsing is now shared with HtmlReporter through EnvironmentFlags, and the docs list all three values.
• Once per run: the docs now say the raise happens at most once and lasts for the rest of the run.
• Process-wide CPU: agreed. The check fails safe: busy processes skip the raise.
• Blocking integration test: left out for the CI timing-flakiness reason given in my earlier comment. Raise/restore is covered deterministically.