Road to BenchmarkDotNet

[adamsitnik]

This is a list of requirements that need to be met before we switch from xunit-performance to BenchmarkDotNet.

Missing features:

• Support netcoreapp2.1 BenchmarkDotNet#587 "Support netcoreapp2.1" (fixed by @eerhardt, was part of 0.10.11 release)
• Per-method parameterization BenchmarkDotNet#256 ParamsSource BenchmarkDotNet#350 generate IParams for users in smart way BenchmarkDotNet#754 "Per-method parameterization" - sth what xunit offers with [InlineData] and [MemberData] (fixed by @adamsitnik, was part of 0.10.14 release)
• BenchmarkSwitcher should support generic types with parameterless public ctors BenchmarkDotNet#652 support generic benchmark types (fixed by @adamsitnik, was part of 0.10.13 release)
• Add .NET Core support for Diagnostics package BenchmarkDotNet#175 "Add .NET Core support for Diagnostics package" - consume the .NET Standard TraceEvent lib and allow to use it in .NET Core apps (fixed by @adamsitnik, will be part of 0.11.00 release)
• Port to .NET Standard 2.0, drop .NET Core 1.1 support BenchmarkDotNet#698 ".NET Standard 2.0, support" (fixed by @adamsitnik, will be part of 0.11.00 release)
• Extend exported json file with FullName using xunit naming convention for integration purpose BenchmarkDotNet#701 "Implement BenchView exporter"
• "Scenarios" support - run given exe as benchmark, gather multiple results BenchmarkDotNet#715 "Scenarios support - run given exe as benchmark, gather multiple results" - we need it for JitBench, the Roslyn team has an initial implementation that we could use

Performance (the tool needs to be fast to run as part of CI):

• Improve Memory Diagnoser BenchmarkDotNet#606 "Improve Memory Diagnoser" - BDN required one extra process run to get the memory statistics, I have changed the architecture to require only one extra iteration. We need one extra iteration because for desktop .NET we are using AppDomain.MonitoringIsEnabled which adds an extra overhead. So we run the benchmarks without overhead, measure time, enable monitoring and run one extra iteration to get the memory statistics (fixed by @adamsitnik, was part of 0.10.12 release)
• Run Disassembly Diagnoser without extra run BenchmarkDotNet#543 "Run Disassembly Diagnoser without extra run" - BDN required one extra process run to get the disassembly, I have changed the architecture to synchronize parent and child processes and get the disassembly after running the benchmarks, but before quiting the process (fixed by @adamsitnik, was part of 0.10.12 release)
• Generate one executable per runtime settings BenchmarkDotNet#699 "Generate one executable per runtime settings" BDN used to build an extra exe per benchmark. It was taking a lot of time. , I have changed the architecture, now it groups the benchmarks by runtime settings (framework/JIT/GC etc) and builds in parallel one exe for the entire group of benchmarks. For BenchmarkDotNet.Samples project with 650 benchmarks it used to take 1h to build the extra exes. Now its's 13s on my PC (fixed by @adamsitnik, will be part of 0.11.00 release)
• Add an optional way to configure storage to remember the PerfectInvocationCount, IterationCount and UnrollFactor BenchmarkDotNet#704 "Add an optional way to configure storage to remember the PerfectInvocationCount, IterationCount and UnrollFactor" By default BDN uses a heuristic to find perfect invocation and iteration count. For CI scenarios we should remember these values, I estimate that it should save us 20-25% of time.
• Allow to optionally run benchmarks in Parallel BenchmarkDotNet#716 Allow to optionally run benchmarks in Parallel.

Private runtimes support (BenchmarkDotNet compiles new exe, so it needs to know how to work with private builds):

• BenchmarkDotNet requires dotnet cli toolchain to be installed BenchmarkDotNet#648 "BenchmarkDotNet requires dotnet cli toolchain to be installed". Prior joining MS I had no idea that you run .NET Core apps without adding dotnet cli to the PATH. Now it's optional and user can provide path to dotnet cli which should be used (fixed by @adamsitnik, was part of 0.10.13 release)
• BenchmarkDotNet should respect LangVersion project setting BenchmarkDotNet#643 "BenchmarkDotNet should respect LangVersion project setting" - just copy it to the auto-generated project (fixed by @adamsitnik, was part of 0.10.13 release)
• Support private builds of .NET Runtime  BenchmarkDotNet#706 "Support private builds of .NET Runtime" - @vitek-karas needed to measure the perf difference after his recent NGEN changes. He wanted to compare the existing .NET Framework with his private build of CLR. This feature simply sends the provided version as COMPLUS_Version env var to the benchmarked process and allows to benchmark private desktop CLR builds. (fixed by @adamsitnik, was part of 0.10.14 release)
• Improve local CoreCLR support BenchmarkDotNet#700 "Support private CoreCLR and CoreFX buids" - users can now use ANY CoreCLR and CoreFX builds for benchmarking. Uses dotnet cli to publish self-contained app, works on Windows, Linux and Mac. (fixed by @adamsitnik, will be part of 0.11.00 release)
• CoreRT support BenchmarkDotNet#718 CoreRT support.

Verification:

• Test BenchmarkDotNet against unstable/multimodal benchmarks from CoreCLR/CoreFX repo  BenchmarkDotNet#714 Test BenchmarkDotNet against unstable/multimodal benchmarks from CoreCLR/CoreFX repo provided by @AndyAyersMS .

I am tagging the Perf Team and the people who are interested in the progress:
@jorive @valenis @adiaaida @DrewScoggins @brianrob
@ViktorHofer @danmosemsft @eerhardt
@AndyAyersMS @JosephTremoulet
@davidfowl @DamianEdwards

[adamsitnik]

Closing as now we have all micro-benchmarks ported to BenchmarkDotNet and working on all OSes and all architectures.