Skip to content

Conversation

@TimothyMakkison
Copy link
Contributor

@TimothyMakkison TimothyMakkison commented Feb 1, 2026

Looks like when I'd run one benchmark it would run other source generators, giving a misleading time. For instance trying to benchmark StaticPropertyInitializationGenerator would also run TestMetadataGenerator and vice versa. You can prove this by changing all the predicates to false in TestMetadataGenerator, low and behold StaticPropertyInitializationGenerator would suddenly run much faster.

Strangely it doesn't happen to all generators, for instance InfrastructureGenerator only runs itself when benchmarked, making it harder for me to spot the issue 😠

Should solve the issue in #4639 / #4624 - I suspect I was consistently getting a faster time on the benchmarks because I was experimenting with TestMetadataGenerator, not expecting it to affect an unrelated source generator

StaticPropertyInitializationGenerator

Method Mean Error StdDev Gen0 Gen1 Allocated
RunGenerator 28.43 ms 0.542 ms 1.201 ms 666.6667 333.3333 8 MB

TestMetadataGenerator

Method Mean Error StdDev Median Gen0 Gen1 Allocated
RunGenerator 201.6 ms 12.64 ms 36.28 ms 191.5 ms 10000.0000 4000.0000 97.07 MB

AotConverter

Method Mean Error StdDev Gen0 Allocated
RunGenerator 60.76 ms 1.212 ms 2.761 ms 1000.0000 9.29 MB

@claude
Copy link
Contributor

claude bot commented Feb 1, 2026

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.


This is a clean, focused fix that correctly addresses the benchmark pollution issue. The single-line change using WithAnalyzerReferences([]) ensures that only the explicitly benchmarked generator runs, not all generators referenced by the test project. The code:

  • Uses modern C# 12 collection expression syntax
  • Correctly applies the Roslyn API to clear analyzer references
  • Aligns with TUnit's "Performance First" principle by improving benchmark accuracy
  • Has no security or logic concerns

@thomhurst thomhurst enabled auto-merge (squash) February 1, 2026 23:07
@thomhurst thomhurst merged commit 7c34fca into thomhurst:main Feb 1, 2026
8 of 10 checks passed
This was referenced Feb 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants