Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Windows/x64: 9 Regressions on 12/25/2022 7:48:20 PM #11132

Closed
performanceautofiler bot opened this issue Dec 27, 2022 · 1 comment
Closed

[Perf] Windows/x64: 9 Regressions on 12/25/2022 7:48:20 PM #11132

performanceautofiler bot opened this issue Dec 27, 2022 · 1 comment
Assignees
@tannergooding
Labels
arch-x64 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr untriaged

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Dec 27, 2022 •
edited
Loading

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 8355deb78ce4e0c14689ce5eac9ea9fc480faeb0
Compare bd6709248295deefa1956da3aeb9b0e086fbaca5
Diff Diff

Regressions in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsPrefix_FirstHalf - Duration of single invocation 6.70 μs 7.21 μs 1.08 0.07 False
IsPrefix_FirstHalf - Duration of single invocation 3.71 μs 4.50 μs 1.21 0.07 False
IsPrefix_FirstHalf - Duration of single invocation 3.73 μs 4.54 μs 1.22 0.08 False
IsPrefix_FirstHalf - Duration of single invocation 3.71 μs 4.52 μs 1.22 0.11 False

graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringSearch*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (pl-PL, None, False))

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.2113108077099595 > 7.169088031841694.
IsChangePoint: Marked as a change because one of 12/23/2022 9:40:21 PM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -3.820433181418042 (T) = (0 -7354.518873031248) / Math.Sqrt((8613.759996065355 / (47)) + (52690.940800709315 / (3))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (47) + (3) - 2, .025) and -0.07434717374211293 = (6845.570084588534 - 7354.518873031248) / 6845.570084588534 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, None, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.496179652149322 > 3.8901280101051294.
IsChangePoint: Marked as a change because one of 11/1/2022 6:58:35 AM, 11/1/2022 4:32:17 PM, 12/23/2022 9:40:21 PM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -22.59696461322638 (T) = (0 -4569.610223041637) / Math.Sqrt((3435.3182118894956 / (47)) + (4054.05945064536 / (3))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (47) + (3) - 2, .025) and -0.22946078554309068 = (3716.759637049424 - 4569.610223041637) / 3716.759637049424 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, IgnoreCase, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.541652772583729 > 3.902149216877937.
IsChangePoint: Marked as a change because one of 11/1/2022 6:58:35 AM, 11/1/2022 4:32:17 PM, 11/18/2022 1:00:00 AM, 11/18/2022 3:21:53 PM, 12/23/2022 9:40:21 PM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -74.30572370416776 (T) = (0 -4527.918905270303) / Math.Sqrt((2838.6943547020524 / (47)) + (164.77439704489302 / (3))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (47) + (3) - 2, .025) and -0.21393161587794024 = (3729.962088511567 - 4527.918905270303) / 3729.962088511567 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, IgnoreCase, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.5240537595785435 > 3.902585473656288.
IsChangePoint: Marked as a change because one of 11/1/2022 6:58:35 AM, 11/1/2022 4:32:17 PM, 11/17/2022 7:01:39 PM, 11/18/2022 3:21:53 PM, 12/23/2022 9:40:21 PM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -14.160563972057991 (T) = (0 -4563.96731122616) / Math.Sqrt((7674.012509234451 / (47)) + (9413.873970413792 / (3))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (47) + (3) - 2, .025) and -0.21694350532787737 = (3750.352659137208 - 4563.96731122616) / 3750.352659137208 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS Windows 10.0.19042
Baseline 8355deb78ce4e0c14689ce5eac9ea9fc480faeb0
Compare bd6709248295deefa1956da3aeb9b0e086fbaca5
Diff Diff

Regressions in System.Collections.ContainsFalse<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableArray - Duration of single invocation 18.14 μs 19.07 μs 1.05 0.03 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsFalse<Int32>.ImmutableArray(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.066574632352946 > 18.87276251657825.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/17/2022 3:19:47 PM, 12/23/2022 9:40:21 PM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -4.019249990180762 (T) = (0 -19912.8004589704) / Math.Sqrt((35158.048807095605 / (47)) + (604663.5986493295 / (3))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (47) + (3) - 2, .025) and -0.09984969009208314 = (18105.019838941113 - 19912.8004589704) / 18105.019838941113 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 8355deb78ce4e0c14689ce5eac9ea9fc480faeb0
Compare bd6709248295deefa1956da3aeb9b0e086fbaca5
Diff Diff

Regressions in System.Text.Json.Reader.Tests.Perf_Base64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadBase64EncodedByteArray_NoEscaping - Duration of single invocation 94.81 ns 129.78 ns 1.37 0.15 False
ReadBase64EncodedByteArray_HeavyEscaping - Duration of single invocation 94.44 ns 128.90 ns 1.36 0.14 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Reader.Tests.Perf_Base64*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Reader.Tests.Perf_Base64.ReadBase64EncodedByteArray_NoEscaping(NumberOfBytes: 100)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 129.78486574630764 > 99.44180681786246.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/14/2022 9:06:25 AM, 11/22/2022 2:20:42 AM, 12/23/2022 9:40:21 PM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -25.55028063016023 (T) = (0 -129.93224427691712) / Math.Sqrt((53.63251079594049 / (47)) + (0.08700826337928207 / (3))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (47) + (3) - 2, .025) and -0.27018474884920257 = (102.29397290011296 - 129.93224427691712) / 102.29397290011296 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Reader.Tests.Perf_Base64.ReadBase64EncodedByteArray_HeavyEscaping(NumberOfBytes: 100)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 128.9025521921711 > 99.19288937337053.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/10/2022 9:13:35 AM, 12/6/2022 2:12:43 AM, 12/17/2022 10:44:50 PM, 12/23/2022 9:40:21 PM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -28.010330174427953 (T) = (0 -128.95267610928155) / Math.Sqrt((42.78397963843383 / (47)) + (0.0050094813851160784 / (3))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (47) + (3) - 2, .025) and -0.2617227141621575 = (102.20365747708055 - 128.95267610928155) / 102.20365747708055 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 8355deb78ce4e0c14689ce5eac9ea9fc480faeb0
Compare bd6709248295deefa1956da3aeb9b0e086fbaca5
Diff Diff

Regressions in System.Diagnostics.Tracing.Perf_EventListener

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteEvent_IntParams - Duration of single invocation 68.70 ns 73.60 ns 1.07 0.01 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Diagnostics.Tracing.Perf_EventListener*'

Payloads

Baseline
Compare

Histogram

System.Diagnostics.Tracing.Perf_EventListener.WriteEvent_IntParams

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 73.5963348983555 > 72.48026579767752.
IsChangePoint: Marked as a change because one of 11/11/2022 5:53:56 PM, 11/18/2022 7:31:41 AM, 12/23/2022 9:40:21 PM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -59.66874064015645 (T) = (0 -73.6334363289504) / Math.Sqrt((0.1409088059576603 / (47)) + (0.006967182574554933 / (3))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (47) + (3) - 2, .025) and -0.062821219343263 = (69.28111237226696 - 73.6334363289504) / 69.28111237226696 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 8355deb78ce4e0c14689ce5eac9ea9fc480faeb0
Compare bd6709248295deefa1956da3aeb9b0e086fbaca5
Diff Diff

Regressions in System.Collections.CreateAddAndRemove<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
List - Duration of single invocation 40.70 μs 45.35 μs 1.11 0.06 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndRemove&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndRemove<String>.List(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 45.352141861761424 > 42.71735537862728.
IsChangePoint: Marked as a change because one of 10/27/2022 8:00:27 PM, 11/10/2022 12:29:27 AM, 12/3/2022 7:24:45 AM, 12/6/2022 2:12:43 AM, 12/23/2022 9:40:21 PM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -41.5431540063143 (T) = (0 -45477.21757492219) / Math.Sqrt((27284.09967984074 / (47)) + (38163.613939340685 / (3))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (47) + (3) - 2, .025) and -0.11776309320080697 = (40685.91801925972 - 45477.21757492219) / 40685.91801925972 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tannergooding tannergooding

Labels
arch-x64 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr untriaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tannergooding @dakersnar