Skip to content

Conversation

@thomhurst
Copy link
Owner

Summary

  • Benchmark projects now reference TUnit.Engine and TUnit.Assertions directly instead of the TUnit metapackage
  • Eliminates ~35-40% overhead in single test execution benchmarks caused by telemetry initialization

Problem

The TUnit metapackage includes Microsoft.Testing.Extensions.Telemetry which adds AppInsights telemetry. During profiling, we found that 91% of single test execution time was spent waiting, primarily due to:

  • AppInsights background thread initialization
  • I/O completion port waiting (GetQueuedCompletionStatus)
  • Thread pool coordination overhead

Changes

Project Before After
TUnit.PerformanceBenchmarks TUnit.csproj TUnit.Engine + TUnit.Assertions
UnifiedTests (speed-comparison) TUnit package TUnit.Engine + TUnit.Assertions

Measured Improvement

  • Before: ~565ms for single test execution
  • After: ~370ms for single test execution
  • Improvement: ~35-40% reduction in startup time

Test plan

  • Build TUnit.PerformanceBenchmarks project and verify tests run
  • Run speed-comparison benchmarks with TestFramework=TUNIT
  • Verify no telemetry-related assemblies in output

🤖 Generated with Claude Code

Benchmark projects now reference TUnit.Engine and TUnit.Assertions
directly instead of the TUnit metapackage, which includes:
- Microsoft.Testing.Extensions.Telemetry (AppInsights)
- Microsoft.Testing.Extensions.CodeCoverage
- Microsoft.Testing.Extensions.TrxReport

This eliminates ~35-40% overhead in single test execution benchmarks
caused by telemetry initialization (AppInsights background threads,
I/O completion port waiting).

Measured improvement:
- Before: ~565ms for single test execution
- After: ~370ms for single test execution

Co-Authored-By: Claude Opus 4.5 <[email protected]>
@thomhurst
Copy link
Owner Author

Summary

Removes telemetry overhead from benchmark projects by switching from TUnit metapackage to direct TUnit.Engine + TUnit.Assertions references.

Critical Issues

Missing Source Generator in UnifiedTests

The tools/speed-comparison/UnifiedTests/UnifiedTests.csproj change removes the TUnit metapackage but fails to add the required source generator reference. The test files use TUnit attributes ([TestClass], [DataDrivenTest], etc.) which require source generation.

Required fix: Add the source generator reference like TUnit.PerformanceBenchmarks does on line 18.

Note: TUnit.PerformanceBenchmarks already has this correctly.

Suggestions

None - the approach is sound once the source generator is added.

Verdict

REQUEST CHANGES - Critical source generator missing in UnifiedTests project

@thomhurst thomhurst merged commit 0c9a91a into main Jan 12, 2026
13 checks passed
@thomhurst thomhurst deleted the perf/remove-telemetry-from-benchmarks branch January 12, 2026 01:39
This was referenced Jan 12, 2026
This was referenced Jan 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants