Skip to content

Conversation

@DustinCampbell
Copy link
Member

I've reworked the three primary tooling serialization benchmarks classes: ProjectRazorJsonSerializationBenchmark, TagHelperResolutionResultSerializationBenchmark, and TagHelperSerializationBenchmark. Each of these benchmarks is now parameterized and runs over two sets of inputs:

  1. A "legacy" set which is the stuff we've been benchmarking against all along. A big problem with these benchmarks is that they come from unknown source (making them super hard to regenerate) and contain data that doesn't include recent changes to our serialization logic. For example, they weren't generated with the tag helper documentation changes that went in earlier this week. So, those changes don't show a meaningful impact to our benchmarks.
  2. A "Telerik" set which is generated using a sample application from one of our partners. This is a good representation of what a super-large Razor project looks like, and that's the one that I expect to improve as we make fixes.

Here's a comparison between the old and new benchmarks:

Old ProjectRazorJsonSerializationBenchmark

BenchmarkDotNet=v0.13.2.2052-nightly, OS=Windows 11 (10.0.25357.1)
Intel Core i7-8700 CPU 3.20GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET SDK=8.0.100-preview.3.23178.7
  [Host]     : .NET 8.0.0 (8.0.23.17408), X64 RyuJIT AVX2
  Job-GQZQXB : .NET 7.0.5 (7.0.523.17405), X64 RyuJIT AVX2
  Job-PWWZLT : .NET Framework 4.8.1 (4.8.9139.0), X64 RyuJIT VectorSize=256

PowerPlanMode=00000000-0000-0000-0000-000000000000  
Method Job Toolchain Mean Error StdDev Median Min Max Gen0 Gen1 Allocated
'Razor ProjectRazorJson Roundtrip JsonConverter Serialization' Job-GQZQXB .NET 7.0 5.197 ms 0.0579 ms 0.0513 ms 5.191 ms 5.088 ms 5.280 ms 23.4375 7.8125 1.88 MB
'Razor ProjectRazorJson Roundtrip JsonConverter Serialization' Job-PWWZLT .NET Framework 4.7.2 7.403 ms 0.3969 ms 1.1515 ms 7.123 ms 5.929 ms 10.642 ms 367.1875 109.3750 2.25 MB

Old TagHelperResolutionResultSerializationBenchmark

BenchmarkDotNet=v0.13.2.2052-nightly, OS=Windows 11 (10.0.25357.1)
Intel Core i7-8700 CPU 3.20GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET SDK=8.0.100-preview.3.23178.7
  [Host]     : .NET 8.0.0 (8.0.23.17408), X64 RyuJIT AVX2
  Job-GQZQXB : .NET 7.0.5 (7.0.523.17405), X64 RyuJIT AVX2
  Job-PWWZLT : .NET Framework 4.8.1 (4.8.9139.0), X64 RyuJIT VectorSize=256

PowerPlanMode=00000000-0000-0000-0000-000000000000  
Method Job Toolchain Mean Error StdDev Median Min Max Gen0 Gen1 Gen2 Allocated
'Razor TagHelperResolutionResult Roundtrip JsonConverter Serialization' Job-GQZQXB .NET 7.0 23.85 ms 0.473 ms 0.485 ms 23.90 ms 22.92 ms 24.86 ms 218.7500 156.2500 156.2500 13.92 MB
'Razor TagHelperResolutionResult Roundtrip JsonConverter Serialization' Job-PWWZLT .NET Framework 4.7.2 31.88 ms 0.400 ms 0.374 ms 31.85 ms 31.39 ms 32.63 ms 1718.7500 750.0000 656.2500 14.26 MB

Old TagHelperSerializationBenchmark

BenchmarkDotNet=v0.13.2.2052-nightly, OS=Windows 11 (10.0.25357.1)
Intel Core i7-8700 CPU 3.20GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET SDK=8.0.100-preview.3.23178.7
  [Host]     : .NET 8.0.0 (8.0.23.17408), X64 RyuJIT AVX2
  Job-GQZQXB : .NET 7.0.5 (7.0.523.17405), X64 RyuJIT AVX2
  Job-PWWZLT : .NET Framework 4.8.1 (4.8.9139.0), X64 RyuJIT VectorSize=256

PowerPlanMode=00000000-0000-0000-0000-000000000000  
Method Job Toolchain Mean Error StdDev Median Min Max Gen0 Gen1 Gen2 Allocated
'Razor TagHelper Roundtrip Serialization' Job-GQZQXB .NET 7.0 24,653.63 μs 344.404 μs 322.156 μs 24,644.51 μs 24,057.60 μs 25,265.87 μs 250.0000 187.5000 187.5000 14583376 B
'Razor TagHelper Serialization' Job-GQZQXB .NET 7.0 13,426.56 μs 263.390 μs 313.547 μs 13,388.31 μs 12,954.45 μs 13,954.28 μs 156.2500 156.2500 156.2500 9201041 B
'Razor TagHelper Deserialization' Job-GQZQXB .NET 7.0 10,598.80 μs 109.242 μs 96.840 μs 10,573.58 μs 10,435.25 μs 10,786.34 μs 62.5000 - - 5366224 B
'TagHelpers GetHashCode' Job-GQZQXB .NET 7.0 17.48 μs 0.121 μs 0.107 μs 17.48 μs 17.33 μs 17.68 μs - - - -
'Razor TagHelper Roundtrip Serialization' Job-PWWZLT .NET Framework 4.7.2 32,565.22 μs 638.653 μs 655.849 μs 32,614.99 μs 31,390.05 μs 33,979.16 μs 1625.0000 750.0000 625.0000 14941334 B
'Razor TagHelper Serialization' Job-PWWZLT .NET Framework 4.7.2 16,021.87 μs 285.355 μs 266.921 μs 16,055.04 μs 15,669.23 μs 16,590.02 μs 1093.7500 968.7500 968.7500 9210338 B
'Razor TagHelper Deserialization' Job-PWWZLT .NET Framework 4.7.2 13,792.37 μs 272.722 μs 424.596 μs 13,721.52 μs 12,997.81 μs 14,765.94 μs 906.2500 93.7500 - 5718967 B
'TagHelpers GetHashCode' Job-PWWZLT .NET Framework 4.7.2 17.68 μs 0.308 μs 0.316 μs 17.66 μs 17.22 μs 18.54 μs - - - -

New ProjectRazorJsonSerializationBenchmark

BenchmarkDotNet=v0.13.2.2052-nightly, OS=Windows 11 (10.0.25357.1)
Intel Core i7-8700 CPU 3.20GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET SDK=8.0.100-preview.3.23178.7
  [Host]     : .NET 8.0.0 (8.0.23.17408), X64 RyuJIT AVX2
  Job-DVIPOB : .NET 7.0.5 (7.0.523.17405), X64 RyuJIT AVX2
  Job-NJNUKQ : .NET Framework 4.8.1 (4.8.9139.0), X64 RyuJIT VectorSize=256

PowerPlanMode=00000000-0000-0000-0000-000000000000  
Method Job Toolchain ResourceSet Mean Error StdDev Median Min Max Gen0 Gen1 Gen2 Allocated
'Serialize ProjectRazorJson' Job-DVIPOB .NET 7.0 Legacy 1.956 ms 0.0377 ms 0.0434 ms 1.951 ms 1.894 ms 2.057 ms 27.3438 25.3906 25.3906 1.1 MB
'Deserialize ProjectRazorJson' Job-DVIPOB .NET 7.0 Legacy 5.499 ms 0.0596 ms 0.0558 ms 5.493 ms 5.410 ms 5.607 ms 23.4375 - - 1.84 MB
'RoundTrip ProjectRazorJson' Job-DVIPOB .NET 7.0 Legacy 3.362 ms 0.0649 ms 0.0820 ms 3.362 ms 3.204 ms 3.503 ms 39.0625 23.4375 23.4375 2.17 MB
'Serialize ProjectRazorJson' Job-NJNUKQ .NET Framework 4.7.2 Legacy 2.102 ms 0.0214 ms 0.0178 ms 2.099 ms 2.075 ms 2.132 ms 70.3125 42.9688 35.1563 1.1 MB
'Deserialize ProjectRazorJson' Job-NJNUKQ .NET Framework 4.7.2 Legacy 6.593 ms 0.0744 ms 0.0696 ms 6.592 ms 6.462 ms 6.730 ms 320.3125 93.7500 - 1.96 MB
'RoundTrip ProjectRazorJson' Job-NJNUKQ .NET Framework 4.7.2 Legacy 4.341 ms 0.0345 ms 0.0306 ms 4.337 ms 4.283 ms 4.391 ms 257.8125 54.6875 31.2500 2.26 MB
'Serialize ProjectRazorJson' Job-DVIPOB .NET 7.0 Telerik 16.044 ms 0.2694 ms 0.2520 ms 16.053 ms 15.566 ms 16.366 ms 140.6250 125.0000 125.0000 9.04 MB
'Deserialize ProjectRazorJson' Job-DVIPOB .NET 7.0 Telerik 15.763 ms 0.1054 ms 0.0934 ms 15.756 ms 15.594 ms 15.913 ms 109.3750 - - 8.36 MB
'RoundTrip ProjectRazorJson' Job-DVIPOB .NET 7.0 Telerik 28.455 ms 0.4165 ms 0.3896 ms 28.463 ms 27.766 ms 28.962 ms 281.2500 156.2500 156.2500 17.39 MB
'Serialize ProjectRazorJson' Job-NJNUKQ .NET Framework 4.7.2 Telerik 19.496 ms 0.1180 ms 0.1104 ms 19.470 ms 19.353 ms 19.733 ms 500.0000 312.5000 312.5000 9.04 MB
'Deserialize ProjectRazorJson' Job-NJNUKQ .NET Framework 4.7.2 Telerik 20.505 ms 0.2335 ms 0.2070 ms 20.469 ms 20.249 ms 21.060 ms 1500.0000 312.5000 - 9.11 MB
'RoundTrip ProjectRazorJson' Job-NJNUKQ .NET Framework 4.7.2 Telerik 36.801 ms 0.2547 ms 0.2258 ms 36.785 ms 36.485 ms 37.148 ms 1928.5714 571.4286 285.7143 18.15 MB

New TagHelperResolutionResultSerializationBenchmark

BenchmarkDotNet=v0.13.2.2052-nightly, OS=Windows 11 (10.0.25357.1)
Intel Core i7-8700 CPU 3.20GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET SDK=8.0.100-preview.3.23178.7
  [Host]     : .NET 8.0.0 (8.0.23.17408), X64 RyuJIT AVX2
  Job-XOQSRT : .NET 7.0.5 (7.0.523.17405), X64 RyuJIT AVX2
  Job-ZVDNIQ : .NET Framework 4.8.1 (4.8.9139.0), X64 RyuJIT VectorSize=256

PowerPlanMode=00000000-0000-0000-0000-000000000000  InvocationCount=1  UnrollFactor=1  
Method Job Toolchain ResourceSet Mean Error StdDev Median Min Max Gen0 Allocated
'RoundTrip TagHelperDescriptorResult' Job-XOQSRT .NET 7.0 Legacy 21.60 ms 0.221 ms 0.207 ms 21.59 ms 21.30 ms 21.91 ms - 15.29 MB
'RoundTrip TagHelperDescriptorResult' Job-ZVDNIQ .NET Framework 4.7.2 Legacy 28.59 ms 0.417 ms 0.369 ms 28.58 ms 27.99 ms 29.16 ms 1000.0000 15.9 MB
'RoundTrip TagHelperDescriptorResult' Job-XOQSRT .NET 7.0 Telerik 24.76 ms 0.389 ms 0.364 ms 24.64 ms 24.39 ms 25.59 ms - 15.9 MB
'RoundTrip TagHelperDescriptorResult' Job-ZVDNIQ .NET Framework 4.7.2 Telerik 32.30 ms 0.414 ms 0.346 ms 32.19 ms 31.80 ms 33.10 ms 1000.0000 16.6 MB

New TagHelperSerializationBenchmark

BenchmarkDotNet=v0.13.2.2052-nightly, OS=Windows 11 (10.0.25357.1)
Intel Core i7-8700 CPU 3.20GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET SDK=8.0.100-preview.3.23178.7
  [Host]     : .NET 8.0.0 (8.0.23.17408), X64 RyuJIT AVX2
  Job-DVIPOB : .NET 7.0.5 (7.0.523.17405), X64 RyuJIT AVX2
  Job-NJNUKQ : .NET Framework 4.8.1 (4.8.9139.0), X64 RyuJIT VectorSize=256

PowerPlanMode=00000000-0000-0000-0000-000000000000  
Method Job Toolchain ResourceSet Mean Error StdDev Median Min Max Gen0 Gen1 Gen2 Allocated
'Serialize Tag Helpers' Job-DVIPOB .NET 7.0 Legacy 12,515.97 μs 244.394 μs 358.229 μs 12,576.41 μs 11,860.88 μs 13,248.55 μs 125.0000 125.0000 125.0000 9200304 B
'Deserialize Tag Helpers' Job-DVIPOB .NET 7.0 Legacy 33,158.12 μs 213.683 μs 189.425 μs 33,179.76 μs 32,865.77 μs 33,543.93 μs 133.3333 66.6667 - 13386763 B
'RoundTrip Tag Helpers' Job-DVIPOB .NET 7.0 Legacy 45,801.81 μs 854.400 μs 799.206 μs 45,889.22 μs 44,456.72 μs 46,945.93 μs 333.3333 166.6667 166.6667 22603993 B
TagHelperDescriptor.GetHashCode() Job-DVIPOB .NET 7.0 Legacy 11.23 μs 0.167 μs 0.156 μs 11.20 μs 10.99 μs 11.53 μs - - - 32 B
'Serialize Tag Helpers' Job-NJNUKQ .NET Framework 4.7.2 Legacy 14,313.92 μs 76.103 μs 63.550 μs 14,342.07 μs 14,189.61 μs 14,372.07 μs 453.1250 328.1250 312.5000 9204928 B
'Deserialize Tag Helpers' Job-NJNUKQ .NET Framework 4.7.2 Legacy 46,612.60 μs 788.227 μs 737.308 μs 46,541.05 μs 45,537.43 μs 47,655.62 μs 2272.7273 727.2727 - 14470011 B
'RoundTrip Tag Helpers' Job-NJNUKQ .NET Framework 4.7.2 Legacy 62,300.40 μs 701.485 μs 621.848 μs 62,238.64 μs 61,452.37 μs 63,543.10 μs 2666.6667 1111.1111 222.2222 23691046 B
TagHelperDescriptor.GetHashCode() Job-NJNUKQ .NET Framework 4.7.2 Legacy 11.93 μs 0.085 μs 0.080 μs 11.94 μs 11.75 μs 12.09 μs - - - 32 B
'Serialize Tag Helpers' Job-DVIPOB .NET 7.0 Telerik 13,764.60 μs 271.426 μs 352.930 μs 13,759.11 μs 13,142.74 μs 14,556.93 μs 156.2500 140.6250 140.6250 9473941 B
'Deserialize Tag Helpers' Job-DVIPOB .NET 7.0 Telerik 42,042.49 μs 436.147 μs 386.633 μs 41,963.93 μs 41,270.62 μs 42,717.51 μs 166.6667 83.3333 - 16242696 B
'RoundTrip Tag Helpers' Job-DVIPOB .NET 7.0 Telerik 53,468.15 μs 887.440 μs 786.692 μs 53,246.33 μs 52,526.90 μs 55,087.91 μs 300.0000 200.0000 100.0000 25711901 B
TagHelperDescriptor.GetHashCode() Job-DVIPOB .NET 7.0 Telerik 10.82 μs 0.209 μs 0.215 μs 10.79 μs 10.44 μs 11.17 μs - - - 32 B
'Serialize Tag Helpers' Job-NJNUKQ .NET Framework 4.7.2 Telerik 16,408.38 μs 243.433 μs 203.278 μs 16,421.66 μs 16,064.45 μs 16,865.87 μs 500.0000 312.5000 312.5000 9478923 B
'Deserialize Tag Helpers' Job-NJNUKQ .NET Framework 4.7.2 Telerik 60,319.55 μs 749.340 μs 700.933 μs 60,490.83 μs 59,103.53 μs 61,352.19 μs 2777.7778 888.8889 - 17661799 B
'RoundTrip Tag Helpers' Job-NJNUKQ .NET Framework 4.7.2 Telerik 72,927.61 μs 723.817 μs 641.644 μs 73,021.11 μs 71,629.36 μs 73,708.00 μs 3000.0000 1000.0000 142.8571 27144155 B
TagHelperDescriptor.GetHashCode() Job-NJNUKQ .NET Framework 4.7.2 Telerik 10.82 μs 0.102 μs 0.085 μs 10.83 μs 10.67 μs 11.01 μs - - - 32 B

@DustinCampbell DustinCampbell requested a review from a team as a code owner May 19, 2023 21:38
@DustinCampbell DustinCampbell merged commit 401e2ff into dotnet:main May 24, 2023
@DustinCampbell DustinCampbell deleted the rework-serialization-benchmarks branch May 24, 2023 16:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants