.NET 7.0 GA Microbenchmarks Performance Study Report #79245
Comments
dakersnar
added
area-Meta
tenet-performance
ghost
added
the
|
Tagging subscribers to this area: @dotnet/area-meta Issue DetailsDataThis time we have covered following configurations comparing .NET 7.0 vs. .NET 6.0:
Most of the benchmarks were run on bare-metal machines, some were executed via WSL. This would not be possible without the help from: @adamsitnik, @jozkee, @janvorli, @jeffhandley, @mrsharm, and @cincuranet who contributed their results and time. The full report generated by the tool is available here. You will have to click "Raw" to see the entire file. The report is sorted from most regressed to most improved, so scroll to the bottom in the full report to see improvements. There are plenty of them! Again, the full historical data turned out to be extremely useful. For details about methodology please read #41871. The RC2 vs 6.0 report can be found here. As that report almost acted as a release-to-release comparison, and contains a different spread of configs, it continues to be a valuable resource when evaluating .NET 7 performance. ImprovementsThere are too many improvements to individually call out (what a great problem to have!). Instead, I collected the top 300 in an excel document (7.0-GA-Report-Top-300.zip). All high-level analysis of this data should be taken with a grain of salt, and I highly encourage readers to examine the raw data for specifics on individual benchmarks (start at the bottom for improvements). Additionally, the previous report was run with a different set of machines, so it may be worth looking at that data as well. The most improved grouping of benchmarks was Just for fun: the winner of the "most improved benchmark" award is System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: None)
For a more in-depth look at .NET 7 performance improvements, check out Stephen Toub's writeup here: https://devblogs.microsoft.com/dotnet/performance_improvements_in_net_7/. To quote his TL;DR:
Thank you to all contributors for making 7.0 the fastest .NET yet! RegressionsFor regressions, this report focused on what was not called out already by the RC2 report. Consider the combination of the two reports to be a total view of .NET 7's regressions compared to .NET 6. Fixed since RC2As part of writing this report, I double checked all regressions called out in the previous RC2 report. Assume all issues in the previous report are unresolved unless called out here:
Investigation In Progress
By Design
Noise, Flaky or MultimodalThe following benchmarks showed up in the report generated by the tool, but were not actual regressions:
StatisticsTotal: 45809 Statistics per Architecture
Statistics per Operating System
Statistics per Namespace
Big thanks to everyone involved!
|
|
Fantastic work on these reports, @dakersnar. And congratulations to the whole team for the exciting performance improvements across the whole product. |
Data
This time we have covered following configurations comparing .NET 7.0 vs. .NET 6.0:
Most of the benchmarks were run on bare-metal machines, some were executed via WSL.
This would not be possible without the help from: @adamsitnik, @jozkee, @janvorli, @jeffhandley, @mrsharm, and @cincuranet who contributed their results and time.
The full report generated by the tool is available here. You will have to click "Raw" to see the entire file. The report is sorted from most regressed to most improved, so scroll to the bottom in the full report to see improvements. There are plenty of them!
Again, the full historical data turned out to be extremely useful. For details about methodology please read #41871.
The RC2 vs 6.0 report can be found here. As that report almost acted as a release-to-release comparison, and contains a different spread of configs, it continues to be a valuable resource when evaluating .NET 7 performance.
Improvements
There are too many improvements to individually call out (what a great problem to have!). Instead, I collected the top 300 in an excel document (7.0-GA-Report-Top-300.zip). All high-level analysis of this data should be taken with a grain of salt, and I highly encourage readers to examine the raw data for specifics on individual benchmarks (start at the bottom for improvements). Additionally, the previous report was run with a different set of machines, so it may be worth looking at that data as well.
The most improved grouping of benchmarks was
System.Text.RegularExpressions.Tests.Perf_Regex*, making up 103 of the top 300, and 56 of the top 100. Feel free to sort the table A to Z in excel to see other groupings of tests.Just for fun: the winner of the "most improved benchmark" award is
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: None), with a staggering 400x improvement.System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: None)
For a more in-depth look at .NET 7 performance improvements, check out Stephen Toub's writeup here: https://devblogs.microsoft.com/dotnet/performance_improvements_in_net_7/. To quote his TL;DR:
Thank you to all contributors for making 7.0 the fastest .NET yet!
Regressions
For regressions, this report focused on what was not called out already by the RC2 report. Consider the combination of the two reports to be a total view of .NET 7's regressions compared to .NET 6.
Fixed since RC2
As part of writing this report, I double checked all regressions called out in the previous RC2 report. Assume all issues in the previous report are unresolved unless called out here:
Benchmark.GetChildKeysTests.AddChainedConfigurationEmptySystem.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 1, 1]),System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 33, 128]),System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 129, 1024]),System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [ 0, 0]),System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 0, 1024]Investigation In Progress
System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite*System.Numerics.Tests.Perf_Matrix4x4.AddOperatorBenchmarkBenchmarksGame.SpectralNorm_3.RunBenchSystem.Tests.Perf_GC<Byte>.AllocateUninitializedArray(length: 10000, pinned: True),System.Tests.Perf_GC<Char>.AllocateUninitializedArray(length: 10000, pinned: True)System.Text.Perf_Utf8Encoding.GetByteCount(Input: Cyrillic),System.Text.Perf_Utf8Encoding.GetByteCount(Input: Chinese)By Design
System.Diagnostics.Perf_Process.GetProcessByIdSystem.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get_EnumerateHeaders_Validated(ssl: True, chunkedResponse: True, responseLength: 100000),System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get_EnumerateHeaders_Unvalidated(ssl: True, chunkedResponse: True, responseLength: 100000),System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get_EnumerateHeaders_Unvalidated(ssl: True, chunkedResponse: False, responseLength: 100000)Noise, Flaky or Multimodal
The following benchmarks showed up in the report generated by the tool, but were not actual regressions:
System.Numerics.Tests.Perf_Matrix4x4.IdentityBenchmarkSystem.Numerics.Tests.Constructor.ConstructorBenchmark_SByteSystem.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 4)Perf_VectorOf*Statistics
Total: 45809
Same: 57.48 %
Slower: 5.99 %
Faster: 24.58 %
Noise: 11.91 %
Unknown: 0.04 %
Statistics per Architecture
Statistics per Operating System
Statistics per Namespace
Big thanks to everyone involved!
The text was updated successfully, but these errors were encountered: