.NET 6.0 Microbenchmarks Performance Study Report

[adamsitnik]

The goal of this study was simple: ensure that we ship .NET 6 without any performance regressions, especially in the configs not covered by .NET Performance Lab.

We have not changed the methodology since last year, so if you are interested in details about methodology please read #41871.

Data

This year, we have covered more configs than ever! Namely:

• architectures: x64, x86, arm64, arm
• Unix: Alpine 3.13, CentOS 8, Debian 10, RHEL 7, SLES 15, OpenSUSE 15.3, Ubuntu 16.04, Ubuntu 18.04, macOS 11.4 and 11.5.2
• Windows: 7, 8.1, 10, 11, Server 2022, Server 2022 Core

Operating System	Bit	Processor Name	Comment
Windows 10.0.19043.1165	X64	AMD Ryzen Threadripper PRO 3945WX 12-Cores
Windows 10.0.20348	X64	AMD EPYC 7452	Windows Server 2022, VM
Windows 10.0.20348	X64	AMD EPYC 7452	Windows Server 2022 Core, VM
Windows 10.0.18363.1621	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
Windows 8.1	X64	Intel Core i7-3610QM CPU 2.30GHz (Ivy Bridge)
Windows 10.0.19042.685	X64	Intel Core i7-5557U CPU 3.10GHz (Broadwell)
Windows 10.0.19043.1165	X64	Intel Core i7-6700 CPU 3.40GHz (Skylake)
Windows 10.0.22454	X64	Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Windows 10.0.22451	X64	Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Windows 10.0.19042.1165	X64	Intel Core i9-9900T CPU 2.10GHz
Windows 7 SP1	X64	Intel Core2 Duo CPU T9600 2.80GHz	ancient hardware
centos 8	X64	AMD EPYC 7452	VM
debian 10	X64	AMD EPYC 7452	VM
rhel 7	X64	AMD EPYC 7452	VM
sles 15	X64	AMD EPYC 7452	VM
opensuse-leap 15.3	X64	AMD EPYC 7452	VM
ubuntu 18.04	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
ubuntu 18.04	X64	Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
alpine 3.13	X64	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
ubuntu 16.04	Arm64	Qualcomm Centriq
Windows 10.0.19043.1165	Arm64	Microsoft SQ1 3.0 GHz
Windows 10.0.22000	Arm64	Microsoft SQ1 3.0 GHz
Windows 10.0.19043.1165	X86	AMD Ryzen Threadripper PRO 3945WX 12-Cores
Windows 10.0.18363.1621	X86	Intel Xeon CPU E5-1650 v4 3.60GHz
Windows 10.0.19043.1165	Arm	Microsoft SQ1 3.0 GHz
macOS Big Sur 11.5.2	X64	Intel Core i5-4278U CPU 2.60GHz (Haswell)
macOS Big Sur 11.5.2	X64	Intel Core i7-4870HQ CPU 2.50GHz (Haswell)
macOS Big Sur 11.4	X64	Intel Core i7-5557U CPU 3.10GHz (Broadwell)

Most of the benchmarks were run on bare-metal machines, but some were executed on Azure VMs.

This would not be possible without the help from: @AndyAyersMS @BruceForstall @bwadswor @carlossanlop @danmoseley @jeffhandley @michaelgsharp @sharwell @smitpatel @vatsan-madhavan @wfurt who contributed their results and time.

Everyone interested can download the data from here and here (GitHub does not support files larger than 100 MB so I had to split .NET 5 and 6 results into two separate archives). The full report generated by the tool is available here. The full report contains also improvements, so if you read it from the end you can see the biggest perf improvements.

Moreover, the full historical data which again turned out to be extremely useful is available here.

Regressions

By design

• System.Memory.Span<Byte>.IndexOfAnyFourValues(Size: 512), System.Memory.Span<Int32>.IndexOfAnyFourValues(Size: 512):
  • reported: [Perf] Regression in Span Indexing #54172
  • explained: [Perf] Regression in Span Indexing #54172 (comment)
• System.Linq.Tests.Perf_Enumerable.TakeLastHalf(input: List)
  • reported: [Perf] Regressions in System.Linq.Tests.Perf_Enumerable #50006
  • explained: [Perf] Regressions in System.Linq.Tests.Perf_Enumerable #50006 (comment)
• System.Collections.Concurrent.AddRemoveFromSameThreads*
  • reported: Regressions in RentReturn ArrayPool and ConcurrentQueue #46714
  • explained: Regressions in RentReturn ArrayPool and ConcurrentQueue #46714 (comment)
• System.Tests.Perf_Random.NextDouble, System.Tests.Perf_Random.Next_int, System.Tests.Perf_Random.Next_int_int, System.Tests.Perf_Random.NextBytes_span
  • reported: [Perf -26%] System.Tests.Perf_Random (3) #47870
  • explained: [Perf -26%] System.Tests.Perf_Random (3) #47870 (comment)
• System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiCharArray(StringLengthInChars: 32)
  • the benchmark has slightly regressed for small inputs (10%), but greatly improved for large inputs (up to few times faster)
• System.IO.Tests.Perf_FileStream.ReadAsync(fileSize: 1024, userBufferSize: 1024, options: Asynchronous)
  • it's the cost of FileStream being 100% async now

Investigation in progress

• System.Numerics.Tests.Perf_Matrix3x2.IsIdentityBenchmark
  • reported: [Perf -1,517%] System.Numerics.Tests.Perf_Vector2.DistanceBenchmark #50939
  • asked for help: [Perf -1,517%] System.Numerics.Tests.Perf_Vector2.DistanceBenchmark #50939 (comment)
  • reopened: [Perf -1,517%] System.Numerics.Tests.Perf_Vector2.DistanceBenchmark #50939 (comment)
• System.Numerics.Tests.Perf_Vector3.DistanceBenchmark, System.Numerics.Tests.Perf_Vector2.DistanceBenchmark
  • detected: [Perf -996%] System.Numerics.Tests.Perf_Vector2.DistanceBenchmark DrewScoggins/performance-2#1848 but not reported in runtime repo
  • added to similar existing issue: [Perf -1,517%] System.Numerics.Tests.Perf_Vector2.DistanceBenchmark #50939 (comment)
• System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Options: (en-US, OrdinalIgnoreCase)), System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (en-US, Ordinal)), System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: 12345), System.Tests.Perf_Int32.ToStringHex(value: 2147483647), System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Options: (en-US, Ordinal))
  • detected: in [Perf] Changes at 3/19/2021 6:47:53 AM DrewScoggins/performance-2#4549 but not reported in runtime repo
  • opened: Potential performance regression in OrdinalIgnoreCase string comparision #59087, most likely caused by PGO
• System.Collections.ContainsKeyFalse<Int32, Int32>.SortedList(Size: 512)
  • reported: [Perf] Regressions in System.Collections.TryGetValueFalse<String, String> #51258
  • added a comment with what I've found: [Perf] Regressions in System.Collections.TryGetValueFalse<String, String> #51258 (comment)
• System.Collections.ContainsKeyFalse<Int32, Int32>.ConcurrentDictionary(Size: 512)
  • detected: [Perf] Changes at 6/28/2021 10:33:32 PM DrewScoggins/performance-2#7188 but not reported in runtime repo
  • opened: Potential performance regression in ConcurrentDictionary #59101
• System.Text.Json.Serialization.Tests.ReadJson<Int32>.DeserializeFromStream
  • seems to not be detected by the bot
  • opened: Performance regression in JsonSerializer.Deserialize(Stream) for very small types like Int32 #59103
• System.Text.Tests.Perf_StringBuilder.ctor_capacity(length: 100000), System.Text.Tests.Perf_StringBuilder.ToString_MultipleSegments(length: 100000) and System.Text.Tests.Perf_StringBuilder.ctor_string(length: 100000)
  • detected (with a monthly delay): [Perf] Changes at 7/8/2021 10:38:40 PM perf-autofiling-issues#304 but not reported in runtime repo
  • opened: Creating StringBuilder with large initial capacity has regressed #59142
• System.Collections.CtorDefaultSize<Int32>.ConcurrentBag
  • seems to affect only Linux VMs (bare metal is fine)
  • opened: Surprising performance regression on Unix VMs #59145
• System.Text.Perf_Utf8Encoding.GetBytes(Input: Cyrillic)
  • detected: [Perf] Regressions in System.Text.Perf_Utf8Encoding for Greek and Cyrillic #52313
• System.Collections.Sort<IntClass>.Array(Size: 512)
  • detected: [Perf] Changes at 5/18/2021 5:35:47 PM DrewScoggins/performance-2#6054 but not reported in runtime repo
  • opened: Potential performance regression in Array.Sort for arrays of reference types #59149
• System.Threading.Tests.Perf_Timer.ShortScheduleAndDisposeWithFiringTimers
  • opened: [macOS] Potential perf regression in Timer  #59150, macOS-specific
• PerfLabTests.DelegatePerf.DelegateInvoke
  • openedhttps://github.com/dotnet/runtime/issues/59152, macOS-specific
• Microsoft.Extensions.Logging.ScopesOverheadBenchmark.FilteredByLevel_InsideScope(HasISupportLoggingScopeLogger: False, CaptureScopes: True)
  • detected: [Perf -13%] Microsoft.Extensions.Logging.ScopesOverheadBenchmark (6) DrewScoggins/performance-2#3245 but not reported in runtime repo
  • FilteredByLevel_InsideScope benchmark has regressed #59267

None of the regressions reported above is critical, but in my opinion, we should have a good understanding of #59145 before we ship .NET 6.

Noise, flaky or multimodal

The following benchmarks showed up in the report generated by the tool, but were not actual regressions:

• System.Collections.CopyTo<Int32>*
• System.Net.Primitives.Tests.IPAddressPerformanceTests.Ctor_Span(address: [16, 32, 48, 64, 80, ...])
• System.Buffers.Tests.ReadOnlySequenceTests<Byte>.SliceTenSegments
• noisy for few configs, very stable and even improving for others: System.Numerics.Tests.Perf_Matrix4x4.CreateReflectionBenchmark
• System.Memory.Span<Int32>.EndsWith(Size: 512)
• Most likely heavily dependent on memory alignment System.Memory.Span<Int32>.BinarySearch(Size: 512) ([Perf] Regressions in System.Memory.Span<Byte>.BinarySearch #56402)
• PerfLabTests.CastingPerf.CheckArrayIsVariantGenericInterfaceNo
• System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "x", comparisonType: InvariantCultureIgnoreCase)
• System.Net.Security.Tests.SslStreamTests.ConcurrentReadWrite
• PerfLabTests.DelegatePerf.MulticastDelegateInvoke(length: 1000)
• System.Numerics.Tests.Perf_BitOperations.PopCount_uint, System.Numerics.Tests.Perf_BitOperations.LeadingZeroCount_uint: memory alignment

Big thanks to everyone involved!

[jeffhandley]

All issues that remain open have been triaged out of 6.0. Thanks for the thorough and actionable report, @adamsitnik!