Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Windows/x64: 53 Regressions on 4/17/2023 1:30:24 AM #16438

Closed
performanceautofiler bot opened this issue Apr 18, 2023 · 4 comments
Closed

[Perf] Windows/x64: 53 Regressions on 4/17/2023 1:30:24 AM #16438

performanceautofiler bot opened this issue Apr 18, 2023 · 4 comments
Assignees
@EgorBo
Labels
arch-x64 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Apr 18, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<StructRecord>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToWriter - Duration of single invocation 377.61 ns 412.74 ns 1.09 0.04 False
SerializeToUtf8Bytes - Duration of single invocation 447.78 ns 497.41 ns 1.11 0.07 False
SerializeToStream - Duration of single invocation 539.55 ns 586.13 ns 1.09 0.07 False
SerializeObjectProperty - Duration of single invocation 685.20 ns 762.34 ns 1.11 0.04 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;StructRecord&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<StructRecord>.SerializeToWriter(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 412.73561890218383 > 395.4339791446013.
IsChangePoint: Marked as a change because one of 3/21/2023 2:16:57 AM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -19.70337826423992 (T) = (0 -416.79880089685213) / Math.Sqrt((13.932873628733942 / (33)) + (16.041725970581506 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.11052875055405259 = (375.3156329261242 - 416.79880089685213) / 375.3156329261242 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<StructRecord>.SerializeToUtf8Bytes(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 497.4104606412133 > 482.42861893067516.
IsChangePoint: Marked as a change because one of 3/20/2023 4:51:16 PM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -17.226015632430627 (T) = (0 -501.3679838339645) / Math.Sqrt((88.20325283066978 / (33)) + (13.923785539539464 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.09317247599159063 = (458.6357549655516 - 501.3679838339645) / 458.6357549655516 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<StructRecord>.SerializeToStream(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 586.1315588324661 > 565.103825928716.
IsChangePoint: Marked as a change because one of 3/17/2023 10:06:19 AM, 4/11/2023 1:13:25 AM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -4.727887858975334 (T) = (0 -569.942125711203) / Math.Sqrt((64.9694215280997 / (25)) + (447.16057984855667 / (12))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (12) - 2, .025) and -0.05526878358423477 = (540.091903197768 - 569.942125711203) / 540.091903197768 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<StructRecord>.SerializeObjectProperty(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 762.3403963446805 > 745.708612154779.
IsChangePoint: Marked as a change because one of 3/20/2023 10:25:17 PM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -16.474958248206487 (T) = (0 -758.7698393828745) / Math.Sqrt((222.33483471856826 / (33)) + (18.541927406356006 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.07900868515209104 = (703.2101315068863 - 758.7698393828745) / 703.2101315068863 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Deep

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteDeepUtf16 - Duration of single invocation 5.03 ms 5.94 ms 1.18 0.01 False Trace Trace
WriteDeepUtf8 - Duration of single invocation 5.29 ms 6.28 ms 1.19 0.01 False 55665185.18518519 61577777.77777778 1.1062170649917498) Trace Trace
WriteDeepUtf16 - Duration of single invocation 5.29 ms 6.14 ms 1.16 0.00 False 55742424.24242424 61616666.666666664 1.1053818972546887) Trace Trace
WriteDeepUtf8 - Duration of single invocation 5.10 ms 5.88 ms 1.15 0.00 True 53822222.22222222 59674603.17460318 1.1087354016751212) Trace Trace

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Deep*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.935777142857144 > 5.259431875000001.
IsChangePoint: Marked as a change because one of 3/17/2023 1:30:52 PM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -20.940851723667183 (T) = (0 -5884508.53139109) / Math.Sqrt((3322192056.517333 / (33)) + (6156688321.132457 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.16837053121081233 = (5036508.859302385 - 5884508.53139109) / 5036508.859302385 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: False, SkipValidation: False)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.2811005 > 5.503290513297872.
IsChangePoint: Marked as a change because one of 3/18/2023 3:39:51 AM, 4/9/2023 4:09:45 AM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -12.665618978253333 (T) = (0 -6150511.333333334) / Math.Sqrt((19912490558.235992 / (33)) + (13774668231.59259 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.150753807247254 = (5344767.312172636 - 6150511.333333334) / 5344767.312172636 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: False)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.14097038327526 > 5.568922073774283.
IsChangePoint: Marked as a change because one of 3/17/2023 1:30:52 PM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -40.7349303341065 (T) = (0 -6159183.613095237) / Math.Sqrt((9676906871.177053 / (33)) + (552825819.7037488 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.15925201596789212 = (5313066.984794295 - 6159183.613095237) / 5313066.984794295 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: False, SkipValidation: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.875389365079365 > 5.2597054028571435.
IsChangePoint: Marked as a change because one of 3/17/2023 1:30:52 PM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -25.198018745752147 (T) = (0 -5878252.33530508) / Math.Sqrt((3574209593.2018104 / (33)) + (3840413033.3756123 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.16294753750920923 = (5054615.230447169 - 5878252.33530508) / 5054615.230447169 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Text.Tests.Utf8FormatterTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FormatterDecimal - Duration of single invocation 38.79 ns 68.65 ns 1.77 0.02 True Trace Trace
FormatterInt32 - Duration of single invocation 2.03 ns 11.71 ns 5.77 0.13 True 33.716870842333066 79.76373439422868 2.3656920823768046) Trace Trace
FormatterInt32 - Duration of single invocation 9.83 ns 14.21 ns 1.45 0.09 False Trace Trace
FormatterUInt32 - Duration of single invocation 2.03 ns 10.15 ns 4.99 0.09 True 32.728533697701515 70.01799189866676 2.1393562126977916) Trace Trace
FormatterInt64 - Duration of single invocation 9.81 ns 16.14 ns 1.64 0.07 False Trace Trace
FormatterUInt64 - Duration of single invocation 2.01 ns 10.46 ns 5.21 0.06 True Trace Trace

graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterDecimal(value: 123456.789)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 68.65015029090281 > 40.72769923050796.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -251.76293144283068 (T) = (0 -68.70228846096629) / Math.Sqrt((0.35207693800458895 / (33)) + (0.01205460891929449 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.7502493345134785 = (39.25285792499024 - 68.70228846096629) / 39.25285792499024 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 4)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.712787950686494 > 1.8069056020683711.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -232.85482992344078 (T) = (0 -11.731360630512757) / Math.Sqrt((0.05458862331473079 / (33)) + (0.0004684325102608711 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -5.074591485591857 = (1.9312180347169061 - 11.731360630512757) / 1.9312180347169061 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 12345)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.209798795450865 > 13.374188392918189.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -10.950455052425918 (T) = (0 -14.209701761366015) / Math.Sqrt((2.2804159929203145 / (33)) + (2.3643506870893198E-07 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.25404494724708104 = (11.331094465600774 - 14.209701761366015) / 11.331094465600774 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt32(value: 0)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.151958075521138 > 0.7288153511036298.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -75.0154722121881 (T) = (0 -10.161148059249276) / Math.Sqrt((0.44831863513940406 / (33)) + (0.0002698784832213175 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -6.279156323244505 = (1.3959238691991978 - 10.161148059249276) / 1.3959238691991978 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: 12345)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.144791345760805 > 13.328661038233225.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -19.04389451896657 (T) = (0 -16.10274540627431) / Math.Sqrt((2.1653788300282204 / (33)) + (0.0008697564337102045 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.4356428710601397 = (11.21640049268197 - 16.10274540627431) / 11.21640049268197 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt64(value: 0)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.461659946371107 > 0.8218326870609063.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -82.95522506538694 (T) = (0 -10.443203143020202) / Math.Sqrt((0.3878079492399679 / (33)) + (0.00036679513187298237 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -6.378364051961744 = (1.4153819287682863 - 10.443203143020202) / 1.4153819287682863 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Basic

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteBasicUtf8 - Duration of single invocation 835.21 ns 938.05 ns 1.12 0.01 True 8845.025279884434 9538.267214358004 1.0783764785896268) Trace Trace
WriteBasicUtf16 - Duration of single invocation 763.47 ns 848.65 ns 1.11 0.00 False 8210.515708059356 8889.189989053424 1.0826591538370591) Trace Trace
WriteBasicUtf8 - Duration of single invocation 1.89 ms 2.59 ms 1.37 0.01 True 18129230.769230768 23997894.736842107 1.3237127952263552) Trace Trace
WriteBasicUtf16 - Duration of single invocation 2.52 ms 3.25 ms 1.29 0.00 True 25181443.29896907 31139473.684210528 1.2366040069468687) Trace Trace
WriteBasicUtf8 - Duration of single invocation 2.62 ms 3.31 ms 1.26 0.00 True Trace Trace
WriteBasicUtf16 - Duration of single invocation 971.11 ns 1.09 μs 1.12 0.01 False Trace Trace
WriteBasicUtf16 - Duration of single invocation 2.62 ms 3.30 ms 1.26 0.00 True 26084397.163120568 32042342.342342343 1.228410307585924) Trace Trace
WriteBasicUtf8 - Duration of single invocation 709.02 ns 805.96 ns 1.14 0.00 True 7352.552285724451 7995.858222771255 1.087494234933335) Trace Trace
WriteBasicUtf8 - Duration of single invocation 671.92 ns 762.34 ns 1.13 0.00 True 6977.509519550975 7616.464778068348 1.0915735416378898) Trace Trace
WriteBasicUtf16 - Duration of single invocation 803.49 ns 917.30 ns 1.14 0.00 False 8585.969717570531 9271.79870453413 1.0798778716351751) Trace Trace
WriteBasicUtf16 - Duration of single invocation 1.89 ms 2.58 ms 1.36 0.00 False 18123076.923076924 24000000 1.3242784380305601) Trace Trace
WriteBasicUtf8 - Duration of single invocation 2.52 ms 3.25 ms 1.29 0.00 True 25160824.74226804 31145614.035087716 1.237861411703478) Trace Trace
WriteBasicUtf16 - Duration of single invocation 2.00 ms 2.71 ms 1.35 0.01 True 19024590.163934425 24905072.463768113 1.3090990265315425) Trace Trace
WriteBasicUtf16 - Duration of single invocation 946.96 ns 1.03 μs 1.09 0.01 False 10050.862645614832 10740.698213427573 1.0686344637406535) Trace Trace
WriteBasicUtf8 - Duration of single invocation 1.98 ms 2.68 ms 1.35 0.00 True Trace Trace
WriteBasicUtf8 - Duration of single invocation 864.87 ns 963.36 ns 1.11 0.01 False Trace Trace

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Basic*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: True, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 938.0515690615598 > 869.0708835126422.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -41.519243407030174 (T) = (0 -938.3895935632188) / Math.Sqrt((64.22620990723605 / (33)) + (19.505707567983855 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.13067122117640237 = (829.9402832477465 - 938.3895935632188) / 829.9402832477465 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: True, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 848.6519746609296 > 792.5702291902551.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -35.484166722572965 (T) = (0 -851.8549642968322) / Math.Sqrt((31.288522006081504 / (33)) + (23.575479122270693 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.12228205299454385 = (759.0382132761181 - 851.8549642968322) / 759.0382132761181 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: True, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.593222395833333 > 1.9360611323529413.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -101.80645994686783 (T) = (0 -2581996.3058035714) / Math.Sqrt((1006340827.0952104 / (33)) + (77806005.57191424 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.38630682990929754 = (1862499.8810490638 - 2581996.3058035714) / 1862499.8810490638 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: True, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.251141666666666 > 2.562094225490196.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -38.28514161412744 (T) = (0 -3235758.5379897878) / Math.Sqrt((2099311522.5903163 / (33)) + (1321254252.11545 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.3069079735068317 = (2475888.5886259177 - 3235758.5379897878) / 2475888.5886259177 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: False, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.3063484444444446 > 2.640586622603587.
IsChangePoint: Marked as a change because one of 2/24/2023 8:06:06 PM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -52.508357784799095 (T) = (0 -3321711.129729729) / Math.Sqrt((3312687040.539375 / (33)) + (425340689.78129315 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.29412650627730624 = (2566759.2106470237 - 3321711.129729729) / 2566759.2106470237 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: False, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0907781107589734 > 1014.3784545588421.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -98.17837371343688 (T) = (0 -1090.7867374173293) / Math.Sqrt((50.622825852852145 / (33)) + (0.11378839085380348 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.1267702990006853 = (968.0648650259336 - 1090.7867374173293) / 968.0648650259336 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: False, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.2992160888888895 > 2.6274957449494956.
IsChangePoint: Marked as a change because one of 2/24/2023 8:06:06 PM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -55.72269316087395 (T) = (0 -3317748.3513678513) / Math.Sqrt((3341699103.2589536 / (32)) + (313414958.3665342 / (4))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (4) - 2, .025) and -0.293774207661904 = (2564395.187135206 - 3317748.3513678513) / 2564395.187135206 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: False, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 805.958004178061 > 743.8247862531567.
IsChangePoint: Marked as a change because one of 3/6/2023 11:14:52 AM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -57.88340601118054 (T) = (0 -804.6846211729951) / Math.Sqrt((30.532872105638674 / (33)) + (6.421053304322842 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.1292134596243686 = (712.6062962804857 - 804.6846211729951) / 712.6062962804857 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: True, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 762.3405728204847 > 690.0848422527354.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -51.95848948214003 (T) = (0 -763.2214133397816) / Math.Sqrt((49.445862306160336 / (33)) + (9.591317614475145 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.15523775138150828 = (660.6617663135332 - 763.2214133397816) / 660.6617663135332 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: False, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 917.3026471549849 > 842.2348443898064.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -17.60219660137803 (T) = (0 -909.1731179827043) / Math.Sqrt((92.31391122115957 / (33)) + (121.48260199252827 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.12549413618122238 = (807.7990713194733 - 909.1731179827043) / 807.7990713194733 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: True, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.5812129464285714 > 1.9442068135465869.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -94.93211974962142 (T) = (0 -2589320.2926065163) / Math.Sqrt((1375718929.083178 / (33)) + (50389956.86147492 / (3))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (3) - 2, .025) and -0.38962605052484367 = (1863321.6408317646 - 2589320.2926065163) / 1863321.6408317646 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: True, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.2478606060606063 > 2.5516279717194568.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -58.441326732165614 (T) = (0 -3221411.5054786485) / Math.Sqrt((2073173106.9807563 / (33)) + (401384362.7211259 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.301635778881854 = (2474894.7115190257 - 3221411.5054786485) / 2474894.7115190257 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: False, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.7108332600732603 > 2.04362365503876.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -74.6959684611207 (T) = (0 -2693699.1678611245) / Math.Sqrt((892061456.3137234 / (33)) + (265232066.59341183 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.36594475492444223 = (1972041.0786380065 - 2693699.1678611245) / 1972041.0786380065 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: True, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.032293302125915 > 973.9294579600211.
IsChangePoint: Marked as a change because one of 2/24/2023 12:59:32 PM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -24.490191392749093 (T) = (0 -1042.5298513178964) / Math.Sqrt((62.046296638416145 / (32)) + (74.63372524629331 / (4))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (4) - 2, .025) and -0.11933558337209531 = (931.3827477700521 - 1042.5298513178964) / 931.3827477700521 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: False, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.6771313664596277 > 2.0508173213675214.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -60.43650095660854 (T) = (0 -2690594.9611801244) / Math.Sqrt((1339795334.080256 / (33)) + (423819359.5498343 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.37348691063988587 = (1958952.022285104 - 2690594.9611801244) / 1958952.022285104 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: False, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 963.3614033699431 > 907.6002329535506.
IsChangePoint: Marked as a change because one of 2/6/2023 2:52:08 AM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -24.22147786615635 (T) = (0 -965.9032818966963) / Math.Sqrt((21.028641656138497 / (33)) + (65.86369638078027 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.115705705411683 = (865.7330308625505 - 965.9032818966963) / 865.7330308625505 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 18, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<SimpleStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToWriter - Duration of single invocation 106.52 ns 113.76 ns 1.07 0.01 False 1153.2011407675436 1211.113795979453 1.0502190408633867) Trace Trace
SerializeToWriter - Duration of single invocation 177.10 ns 188.27 ns 1.06 0.03 False Trace Trace

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;SimpleStructWithProperties&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<SimpleStructWithProperties>.SerializeToWriter(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 113.76187674558125 > 111.11888441759973.
IsChangePoint: Marked as a change because one of 2/4/2023 8:08:12 PM, 3/20/2023 4:51:16 PM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -20.541273070009765 (T) = (0 -114.18638410910569) / Math.Sqrt((3.8688384754995724 / (31)) + (0.09227016439539344 / (4))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (4) - 2, .025) and -0.07431618408760297 = (106.2875025066127 - 114.18638410910569) / 106.2875025066127 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Serialization.Tests.WriteJson<SimpleStructWithProperties>.SerializeToWriter(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 188.26624489530852 > 187.79061920548688.
IsChangePoint: Marked as a change because one of 3/20/2023 4:51:16 PM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -14.946330228044971 (T) = (0 -189.52028298784148) / Math.Sqrt((1.9690191515920914 / (33)) + (1.9685305411719656 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.06222824064112123 = (178.41766556070297 - 189.52028298784148) / 178.41766556070297 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToWriter - Duration of single invocation 336.66 ns 376.17 ns 1.12 0.01 False 3647.6241577422015 3938.4943386931004 1.0797423660915058) Trace Trace
SerializeToUtf8Bytes - Duration of single invocation 437.34 ns 465.52 ns 1.06 0.07 False Trace Trace

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;LargeStructWithProperties&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.SerializeToWriter(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 376.1653778032429 > 353.8017278843143.
IsChangePoint: Marked as a change because one of 2/20/2023 11:07:30 PM, 3/20/2023 8:31:51 AM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -15.220847989446426 (T) = (0 -375.78158944535846) / Math.Sqrt((21.051791788196038 / (33)) + (19.6784898053216 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.10556757572005138 = (339.8992496687627 - 375.78158944535846) / 339.8992496687627 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.SerializeToUtf8Bytes(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 465.5177845774972 > 448.2146404931039.
IsChangePoint: Marked as a change because one of 3/23/2023 12:23:24 PM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -23.198332959042393 (T) = (0 -464.5329004625681) / Math.Sqrt((60.570586077474154 / (32)) + (2.473228675496765 / (4))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (4) - 2, .025) and -0.08593700640046374 = (427.77149846135836 - 464.5329004625681) / 427.77149846135836 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.MathBenchmarks.Double

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ScaleB - Duration of single invocation 11.65 μs 13.48 μs 1.16 0.02 False Trace Trace

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.MathBenchmarks.Double*'

Payloads

Baseline
Compare

Histogram

System.MathBenchmarks.Double.ScaleB

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.478316810344829 > 12.233978374359538.
IsChangePoint: Marked as a change because one of 2/24/2023 9:01:57 AM, 3/6/2023 11:14:52 AM, 3/20/2023 8:31:51 AM, 3/22/2023 7:51:01 PM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -152.01828047448814 (T) = (0 -13485.220227719321) / Math.Sqrt((331.3301230667949 / (33)) + (533.7092249930316 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.15610299479654358 = (11664.376174453655 - 13485.220227719321) / 11664.376174453655 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<ClassRecord>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeObjectProperty - Duration of single invocation 700.50 ns 738.39 ns 1.05 0.07 False
SerializeToUtf8Bytes - Duration of single invocation 443.99 ns 489.37 ns 1.10 0.04 False
SerializeToString - Duration of single invocation 485.63 ns 547.60 ns 1.13 0.08 False
SerializeToWriter - Duration of single invocation 361.86 ns 412.03 ns 1.14 0.03 False
SerializeToStream - Duration of single invocation 560.36 ns 613.67 ns 1.10 0.07 False

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;ClassRecord&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<ClassRecord>.SerializeObjectProperty(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 738.3885971664586 > 730.7372316185857.
IsChangePoint: Marked as a change because one of 2/24/2023 9:01:57 AM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -11.517992837809885 (T) = (0 -745.9104033243804) / Math.Sqrt((454.87975296858133 / (33)) + (32.43680918502067 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.07787837363659515 = (692.0172271457621 - 745.9104033243804) / 692.0172271457621 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<ClassRecord>.SerializeToUtf8Bytes(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 489.36662041160383 > 467.66839339041235.
IsChangePoint: Marked as a change because one of 3/20/2023 10:25:17 PM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -16.84176389487092 (T) = (0 -491.1536037472586) / Math.Sqrt((119.74817853946429 / (32)) + (6.257834668544678 / (3))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (3) - 2, .025) and -0.09025265262891909 = (450.4952155474634 - 491.1536037472586) / 450.4952155474634 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<ClassRecord>.SerializeToString(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 547.5973759207185 > 511.7798435348854.
IsChangePoint: Marked as a change because one of 3/25/2023 12:18:49 AM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -10.865714250290017 (T) = (0 -537.3694050062873) / Math.Sqrt((133.1385736168791 / (33)) + (56.94271268489968 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.09460502352879427 = (490.9253963351205 - 537.3694050062873) / 490.9253963351205 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<ClassRecord>.SerializeToWriter(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 412.027918646948 > 379.7484517304397.
IsChangePoint: Marked as a change because one of 3/20/2023 10:25:17 PM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -44.87287587247324 (T) = (0 -414.04569091127155) / Math.Sqrt((24.913422741425027 / (33)) + (1.9097409477772282 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.1367667865679367 = (364.23098898001354 - 414.04569091127155) / 364.23098898001354 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<ClassRecord>.SerializeToStream(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 613.6743233968186 > 583.2965252192517.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -19.442454119307037 (T) = (0 -612.0603242816271) / Math.Sqrt((129.56749785587255 / (33)) + (6.862441138286798 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.08160934750371703 = (565.8792850618589 - 612.0603242816271) / 565.8792850618589 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Double

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToStringWithFormat - Duration of single invocation 233.46 ns 258.34 ns 1.11 0.00 False Trace Trace
ToString - Duration of single invocation 241.32 ns 266.36 ns 1.10 0.00 False 1911.3897903382745 1966.309490185002 1.028732862404276) Trace Trace
ToStringWithFormat - Duration of single invocation 230.67 ns 246.04 ns 1.07 0.00 False 1736.420534788341 1784.7727946749267 1.0278459387676382) Trace Trace
ToStringWithCultureInfo - Duration of single invocation 239.07 ns 264.07 ns 1.10 0.00 False Trace Trace
ToString - Duration of single invocation 247.02 ns 271.84 ns 1.10 0.01 False 1932.9386911032705 1983.8764184838676 1.0263524795768475) Trace Trace
ToStringWithFormat - Duration of single invocation 244.62 ns 259.97 ns 1.06 0.00 False 1916.5968012382305 1967.7549724315118 1.0266921927242236) Trace Trace

graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Double*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Double.ToStringWithFormat(value: -1.7976931348623157E+308, format: "G17")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 258.3423087963641 > 245.50668425173555.
IsChangePoint: Marked as a change because one of 3/21/2023 12:39:01 PM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -7.696695113132841 (T) = (0 -254.20422550351253) / Math.Sqrt((2.1706500210764865 / (33)) + (24.384998233411817 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.0812674109855015 = (235.09838817006676 - 254.20422550351253) / 235.09838817006676 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Tests.Perf_Double.ToString(value: 1.7976931348623157E+308)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 266.36355712226003 > 253.8348246420276.
IsChangePoint: Marked as a change because one of 3/4/2023 9:40:21 AM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -10.445170982832165 (T) = (0 -263.2268640826073) / Math.Sqrt((4.027831462371518 / (33)) + (13.597210053599726 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.08045381484430475 = (243.62620638304537 - 263.2268640826073) / 243.62620638304537 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Tests.Perf_Double.ToStringWithFormat(value: 1.7976931348623157E+308, format: "G17")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 246.04123458576507 > 241.26227888454144.
IsChangePoint: Marked as a change because one of 2/9/2023 1:20:08 AM, 3/27/2023 2:09:01 AM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -18.33917429851822 (T) = (0 -245.94718057916887) / Math.Sqrt((1.9379897198984941 / (33)) + (2.5597165032896068 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.06646879503448491 = (230.61826255424194 - 245.94718057916887) / 230.61826255424194 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Tests.Perf_Double.ToStringWithCultureInfo(value: -1.7976931348623157E+308, culture: zh)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 264.06632095129845 > 251.26820382843255.
IsChangePoint: Marked as a change because one of 2/17/2023 5:12:50 PM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -11.374609900373404 (T) = (0 -260.6532225333318) / Math.Sqrt((3.494134832056713 / (33)) + (11.823019655327167 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.08266968727994216 = (240.75045749935697 - 260.6532225333318) / 240.75045749935697 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Tests.Perf_Double.ToString(value: -1.7976931348623157E+308)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 271.8379123557325 > 255.66372562604707.
IsChangePoint: Marked as a change because one of 3/8/2023 4:46:51 PM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -6.911474119944433 (T) = (0 -267.9702930873741) / Math.Sqrt((6.004831245243903 / (33)) + (27.91208749305499 / (3))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (3) - 2, .025) and -0.08629232339469117 = (246.6834086150587 - 267.9702930873741) / 246.6834086150587 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Tests.Perf_Double.ToStringWithFormat(value: 1.7976931348623157E+308, format: "G")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 259.96769466522727 > 257.5290148108767.
IsChangePoint: Marked as a change because one of 2/17/2023 5:12:50 PM, 3/25/2023 12:18:49 AM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -6.254420025401722 (T) = (0 -264.7994445598395) / Math.Sqrt((3.093325268102065 / (33)) + (37.14045608187008 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.07797463246203583 = (245.64533949658156 - 264.7994445598395) / 245.64533949658156 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IterateForEach<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ConcurrentBag - Duration of single invocation 4.82 μs 5.15 μs 1.07 0.02 False Trace Trace

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateForEach&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.IterateForEach<String>.ConcurrentBag(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.150228277769281 > 5.045581726808786.
IsChangePoint: Marked as a change because one of 2/21/2023 5:46:41 AM, 3/14/2023 8:42:09 PM, 3/20/2023 8:31:51 AM, 3/25/2023 12:18:49 AM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -16.1881852882872 (T) = (0 -5208.948076710824) / Math.Sqrt((745.7232602828987 / (33)) + (2824.694735766185 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.09157970467832521 = (4771.9356217289105 - 5208.948076710824) / 4771.9356217289105 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Int32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryFormat - Duration of single invocation 15.99 ns 18.21 ns 1.14 0.29 False 215.1333429672561 236.59943286927276 1.0997803948283547) Trace Trace
ToString - Duration of single invocation 13.07 ns 14.08 ns 1.08 0.05 False 152.02673586463774 160.6081376262018 1.0564466619161303) Trace Trace

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int32*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int32.TryFormat(value: -2147483648)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.20899720222771 > 16.72325569319056.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -11.240927632207708 (T) = (0 -18.207629324647648) / Math.Sqrt((0.5451361839706345 / (33)) + (0.04684984771726348 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.11573913718567017 = (16.31889454964751 - 18.207629324647648) / 16.31889454964751 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Tests.Perf_Int32.ToString(value: 2147483647)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.076533095335556 > 13.73125238622533.
IsChangePoint: Marked as a change because one of 2/9/2023 1:20:08 AM, 3/6/2023 11:14:52 AM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -5.112355319157716 (T) = (0 -14.46040907392754) / Math.Sqrt((0.07165557154337984 / (33)) + (0.3158010870913188 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.11197006850284602 = (13.004315029267831 - 14.46040907392754) / 13.004315029267831 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsKeyFalse<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableDictionary - Duration of single invocation 24.70 μs 26.24 μs 1.06 0.02 False Trace Trace

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsKeyFalse&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyFalse<Int32, Int32>.ImmutableDictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 26.238121248499397 > 25.990251562499996.
IsChangePoint: Marked as a change because one of 2/9/2023 12:40:40 PM, 2/13/2023 6:54:14 PM, 3/14/2023 8:42:09 PM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -4.335708127357437 (T) = (0 -27373.391837910793) / Math.Sqrt((40795.90097150998 / (33)) + (1395858.2490385307 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.10342691359230521 = (24807.616617573938 - 27373.391837910793) / 24807.616617573938 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Encodings.Web.Tests.Perf_Encoders

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
EncodeUtf16 - Duration of single invocation 72.43 ns 79.55 ns 1.10 0.37 False Trace Trace

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Encodings.Web.Tests.Perf_Encoders*'

Payloads

Baseline
Compare

Histogram

System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf16(arguments: JavaScript,no escaping required,512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 79.55344665850036 > 72.40185600948736.
IsChangePoint: Marked as a change because one of 4/3/2023 7:13:05 PM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -10.06726058714403 (T) = (0 -77.51106129086563) / Math.Sqrt((2.119502796421673 / (33)) + (2.45573759531643 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.11976847228710037 = (69.22061409047456 - 77.51106129086563) / 69.22061409047456 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Globalization.Tests.Perf_NumberCultureInfo

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 135.09 ns 141.90 ns 1.05 0.00 False Trace Trace

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.Perf_NumberCultureInfo*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.Perf_NumberCultureInfo.ToString(culturestring: )

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 141.89763133428326 > 141.88661931170824.
IsChangePoint: Marked as a change because one of 2/17/2023 5:12:50 PM, 3/15/2023 12:03:45 AM, 4/6/2023 1:55:21 PM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -8.614681983937457 (T) = (0 -143.71195181440336) / Math.Sqrt((0.7951004669795421 / (33)) + (3.115020276017935 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.05675962474466022 = (135.99303800911943 - 143.71195181440336) / 135.99303800911943 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 18, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Numerics.Tests.Perf_BigInteger

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Remainder - Duration of single invocation 8.32 ns 10.12 ns 1.22 0.17 False Trace Trace

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BigInteger*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_BigInteger.Remainder(arguments: 16,8 bits)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.117187170297294 > 8.739444593066253.
IsChangePoint: Marked as a change because one of 2/24/2023 9:01:57 AM, 4/12/2023 8:39:35 PM, 4/18/2023 3:11:52 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 3:11:52 AM.
IsRegressionStdDev: Marked as regression because -19.543341820418426 (T) = (0 -10.044719216270991) / Math.Sqrt((0.11591782666887679 / (32)) + (0.008083943778483598 / (4))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (4) - 2, .025) and -0.17118127538422342 = (8.576570875397296 - 10.044719216270991) / 8.576570875397296 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@EgorBo EgorBo mentioned this issue Apr 18, 2023
Closed
@EgorBo EgorBo closed this as completed Apr 18, 2023
@EgorBo EgorBo removed the untriaged label Apr 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@EgorBo EgorBo

Labels
arch-x64 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@EgorBo