Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 22 Regressions on 1/20/2023 12:33:18 PM #12060

Closed
performanceautofiler bot opened this issue Jan 24, 2023 · 8 comments
Closed

[Perf] Linux/x64: 22 Regressions on 1/20/2023 12:33:18 PM #12060

performanceautofiler bot opened this issue Jan 24, 2023 · 8 comments
Labels
arch-x64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-aot os-linux perf-regression runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Jan 24, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b5fb7d002acbf729151da9d9918a93c8261d1e95
Compare 225ae771d66dd2732fef7d8ad3a5f2b7b00f8c05
Diff Diff

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Interpolation_MultipleArgs - Duration of single invocation 316.48 ns 365.02 ns 1.15 0.05 False
ToLower - Duration of single invocation 32.45 ns 37.80 ns 1.16 0.16 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.Interpolation_MultipleArgs(c: '1', s: "Foo")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 365.02427205165696 > 322.4441102204236.
IsChangePoint: Marked as a change because one of 12/6/2022 2:12:43 AM, 1/20/2023 8:02:54 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -44.26739388370965 (T) = (0 -368.89913979434965) / Math.Sqrt((32.23272168510392 / (41)) + (25.11617636126115 / (22))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (22) - 2, .025) and -0.1999223679317683 = (307.4358388953097 - 368.89913979434965) / 307.4358388953097 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToLower(s: "test")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 37.79884320591748 > 34.39615915502652.
IsChangePoint: Marked as a change because one of 1/20/2023 12:33:18 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -7.887207975198271 (T) = (0 -35.84727810682467) / Math.Sqrt((1.3065994287938771 / (42)) + (0.7707673930413872 / (21))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (21) - 2, .025) and -0.06077811686305898 = (33.79338010179971 - 35.84727810682467) / 33.79338010179971 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline b5fb7d002acbf729151da9d9918a93c8261d1e95
Compare 225ae771d66dd2732fef7d8ad3a5f2b7b00f8c05
Diff Diff

Regressions in PerfLabTests.LowLevelPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
StructWithInterfaceInterfaceMethod - Duration of single invocation 152.30 μs 217.64 μs 1.43 0.06 False
StaticDelegate - Duration of single invocation 174.31 μs 217.56 μs 1.25 0.01 False
GenericClassGenericStaticField - Duration of single invocation 65.24 μs 70.54 μs 1.08 0.04 False
IntegerFormatting - Duration of single invocation 6.92 ms 11.85 ms 1.71 0.01 False
SealedClassInterfaceMethod - Duration of single invocation 152.25 μs 195.82 μs 1.29 0.03 False
EmptyStaticFunction - Duration of single invocation 1.87 ms 2.18 ms 1.16 0.03 False
ClassVirtualMethod - Duration of single invocation 152.21 μs 195.88 μs 1.29 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.LowLevelPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.LowLevelPerf.StructWithInterfaceInterfaceMethod

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 217.63896601190476 > 160.01084796344054.
IsChangePoint: Marked as a change because one of 11/11/2022 11:24:34 AM, 11/21/2022 6:40:27 PM, 11/25/2022 11:21:20 AM, 12/20/2022 9:29:18 AM, 1/20/2023 8:02:54 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -28.837405648161067 (T) = (0 -218743.7638890358) / Math.Sqrt((117995090.40503362 / (41)) + (21948913.76153468 / (22))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (22) - 2, .025) and -0.3504966584776179 = (161972.8286744673 - 218743.7638890358) / 161972.8286744673 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.LowLevelPerf.StaticDelegate

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 217.5576684283088 > 183.0527908472222.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 12/15/2022 2:06:03 PM, 1/17/2023 5:52:57 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -1303.774969729699 (T) = (0 -217578.65911911675) / Math.Sqrt((12238.135660497319 / (23)) + (22616.118018423775 / (40))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (40) - 2, .025) and -0.24768003706409908 = (174386.58362371373 - 217578.65911911675) / 174386.58362371373 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.LowLevelPerf.GenericClassGenericStaticField

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 70.54188383091518 > 68.5407966008113.
IsChangePoint: Marked as a change because one of 11/8/2022 11:25:18 AM, 12/15/2022 2:06:03 PM, 12/20/2022 9:29:18 AM, 1/17/2023 5:52:57 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -63.02369789040199 (T) = (0 -71902.32739163729) / Math.Sqrt((652.9038226330102 / (23)) + (440882.5550677073 / (40))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (40) - 2, .025) and -0.10149212136734667 = (65277.20534431124 - 71902.32739163729) / 65277.20534431124 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.LowLevelPerf.IntegerFormatting

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.85258678125 > 7.282933068177084.
IsChangePoint: Marked as a change because one of 1/15/2023 4:08:29 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -246.2496587789677 (T) = (0 -11904643.46416796) / Math.Sqrt((327702090.5187984 / (9)) + (20002890170.524975 / (54))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (9) + (54) - 2, .025) and -0.7159230129732098 = (6937749.172989164 - 11904643.46416796) / 6937749.172989164 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.LowLevelPerf.SealedClassInterfaceMethod

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 195.81585647321427 > 159.94201470488704.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 11:25:18 AM, 12/15/2022 2:06:03 PM, 12/20/2022 9:29:18 AM, 1/17/2023 5:52:57 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -342.49122830103204 (T) = (0 -195864.57483568238) / Math.Sqrt((285659.17487788346 / (23)) + (145369.77914543005 / (40))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (40) - 2, .025) and -0.2846177055567522 = (152469.15404361085 - 195864.57483568238) / 152469.15404361085 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.LowLevelPerf.EmptyStaticFunction

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.1771842742187504 > 1.9672811283253204.
IsChangePoint: Marked as a change because one of 11/8/2022 11:25:18 AM, 12/15/2022 2:06:03 PM, 1/3/2023 6:29:42 AM, 1/17/2023 5:52:57 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -74.27846701057172 (T) = (0 -2176858.131304614) / Math.Sqrt((54510259.39095726 / (23)) + (545728827.2243725 / (39))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (39) - 2, .025) and -0.1601295705279916 = (1876392.246698698 - 2176858.131304614) / 1876392.246698698 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.LowLevelPerf.ClassVirtualMethod

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 195.8752615985577 > 159.90263282471102.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 11:25:18 AM, 1/17/2023 5:52:57 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -119.06816362740898 (T) = (0 -195803.44618927655) / Math.Sqrt((3017725.947843175 / (23)) + (5118.16725730848 / (39))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (39) - 2, .025) and -0.28267301465036804 = (152652.65890282163 - 195803.44618927655) / 152652.65890282163 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b5fb7d002acbf729151da9d9918a93c8261d1e95
Compare 225ae771d66dd2732fef7d8ad3a5f2b7b00f8c05
Diff Diff

Regressions in System.Collections.Tests.Perf_PriorityQueue<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Enumerate - Duration of single invocation 105.49 ns 114.58 ns 1.09 0.02 False
Enumerate - Duration of single invocation 6.87 μs 7.43 μs 1.08 0.06 False
Enumerate - Duration of single invocation 721.02 ns 775.72 ns 1.08 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_PriorityQueue&lt;String, String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_PriorityQueue<String, String>.Enumerate(Size: 10)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 114.58133967539553 > 110.78841947060363.
IsChangePoint: Marked as a change because one of 12/10/2022 11:25:25 PM, 12/17/2022 10:44:50 PM, 1/20/2023 8:02:54 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -18.6477793840015 (T) = (0 -114.96935658625725) / Math.Sqrt((5.229147025782363 / (41)) + (0.869404170490793 / (22))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (22) - 2, .025) and -0.07100183640205365 = (107.34748781803003 - 114.96935658625725) / 107.34748781803003 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_PriorityQueue&lt;String, String&gt;.Enumerate(Size: 1000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.434540140214796 > 7.218046196359217.
IsChangePoint: Marked as a change because one of 12/13/2022 8:57:50 AM, 12/20/2022 9:29:18 AM, 1/18/2023 3:20:52 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -8.966001788891642 (T) = (0 -7362.639190659966) / Math.Sqrt((510.18703413639906 / (29)) + (91515.17262774732 / (33))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (33) - 2, .025) and -0.06875533064397213 = (6888.98476531916 - 7362.639190659966) / 6888.98476531916 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_PriorityQueue&lt;String, String&gt;.Enumerate(Size: 100)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 775.7179990994563 > 757.1601731532279.
IsChangePoint: Marked as a change because one of 12/13/2022 8:57:50 AM, 12/20/2022 9:29:18 AM, 1/7/2023 8:10:22 PM, 1/11/2023 4:46:06 PM, 1/18/2023 3:20:52 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -39.47850001014848 (T) = (0 -769.8197311579314) / Math.Sqrt((11.91154728551008 / (29)) + (33.32199516200401 / (34))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (34) - 2, .025) and -0.06437210233176202 = (723.2618456190809 - 769.8197311579314) / 723.2618456190809 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b5fb7d002acbf729151da9d9918a93c8261d1e95
Compare 225ae771d66dd2732fef7d8ad3a5f2b7b00f8c05
Diff Diff

Regressions in System.Buffers.Text.Tests.Utf8ParserTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParseUInt64Hex - Duration of single invocation 43.86 ns 49.15 ns 1.12 0.07 False
TryParseUInt32Hex - Duration of single invocation 31.75 ns 34.09 ns 1.07 0.09 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8ParserTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Text.Tests.Utf8ParserTests.TryParseUInt64Hex(value: FFFFFFFFFFFFFFFF)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 49.14856180979969 > 46.082652973837845.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/10/2022 11:11:41 PM, 11/29/2022 11:48:02 PM, 12/17/2022 10:44:50 PM, 12/28/2022 5:13:50 PM, 1/20/2023 8:02:54 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -43.76853388917813 (T) = (0 -48.80091523156138) / Math.Sqrt((0.15665278506870203 / (41)) + (0.18376597606387698 / (22))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (22) - 2, .025) and -0.10982510645397303 = (43.97171675768471 - 48.80091523156138) / 43.97171675768471 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8ParserTests.TryParseUInt32Hex(value: FFFFFFFFFFFFFFFF)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.089720557438326 > 33.36844359670665.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/28/2022 5:13:50 PM, 1/20/2023 4:48:32 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -49.46115248966678 (T) = (0 -34.09128009864793) / Math.Sqrt((0.07779288203026 / (40)) + (0.0027144235249025985 / (23))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (23) - 2, .025) and -0.07054368420247854 = (31.844828568621057 - 34.09128009864793) / 31.844828568621057 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b5fb7d002acbf729151da9d9918a93c8261d1e95
Compare 225ae771d66dd2732fef7d8ad3a5f2b7b00f8c05
Diff Diff

Regressions in System.Collections.Tests.Add_Remove_SteadyState<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableQueue - Duration of single invocation 92.21 ns 106.51 ns 1.16 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Add_Remove_SteadyState&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Add_Remove_SteadyState<String>.ImmutableQueue(Count: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 106.50960329695447 > 96.8125178007926.
IsChangePoint: Marked as a change because one of 12/14/2022 12:26:48 AM, 1/9/2023 10:41:51 AM, 1/11/2023 4:46:06 PM, 1/20/2023 8:02:54 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -53.34459654097049 (T) = (0 -105.03852976302109) / Math.Sqrt((0.4295855298436509 / (41)) + (1.0182225881302591 / (22))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (22) - 2, .025) and -0.1376491051041724 = (92.32946195075054 - 105.03852976302109) / 92.32946195075054 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b5fb7d002acbf729151da9d9918a93c8261d1e95
Compare 225ae771d66dd2732fef7d8ad3a5f2b7b00f8c05
Diff Diff

Regressions in System.Collections.IterateForEach<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableArray - Duration of single invocation 1.76 μs 1.98 μs 1.13 0.06 False
Stack - Duration of single invocation 5.03 μs 5.70 μs 1.13 0.04 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateForEach&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.IterateForEach<String>.ImmutableArray(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.978978064751704 > 1.8462438596521475.
IsChangePoint: Marked as a change because one of 11/25/2022 11:21:20 AM, 12/8/2022 9:08:34 PM, 1/11/2023 4:46:06 PM, 1/20/2023 8:02:54 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -35.21154589542926 (T) = (0 -1973.5194249297483) / Math.Sqrt((1372.8237629099874 / (41)) + (140.45375855539564 / (22))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (22) - 2, .025) and -0.12695876023165592 = (1751.190455739547 - 1973.5194249297483) / 1751.190455739547 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.IterateForEach&lt;String&gt;.Stack(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.696034148353311 > 5.2801821886750195.
IsChangePoint: Marked as a change because one of 11/8/2022 11:25:18 AM, 11/21/2022 6:40:27 PM, 12/10/2022 11:25:25 PM, 1/3/2023 6:29:42 AM, 1/11/2023 4:46:06 PM, 1/20/2023 8:02:54 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -17.816498051674223 (T) = (0 -5530.77261750284) / Math.Sqrt((48310.6189655136 / (40)) + (7626.145392422686 / (22))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (22) - 2, .025) and -0.14548127888052043 = (4828.339597926967 - 5530.77261750284) / 4828.339597926967 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b5fb7d002acbf729151da9d9918a93c8261d1e95
Compare 225ae771d66dd2732fef7d8ad3a5f2b7b00f8c05
Diff Diff

Regressions in System.Collections.Sort<IntClass>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Array_ComparerStruct - Duration of single invocation 59.77 μs 64.46 μs 1.08 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Sort&lt;IntClass&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Sort<IntClass>.Array_ComparerStruct(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 64.46091249333332 > 63.03821861588462.
IsChangePoint: Marked as a change because one of 11/25/2022 7:59:25 AM, 11/29/2022 11:48:02 PM, 12/14/2022 12:26:48 AM, 1/20/2023 8:02:54 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -13.339304951703568 (T) = (0 -63550.99986292539) / Math.Sqrt((946649.0993477035 / (41)) + (629191.0949968678 / (22))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (22) - 2, .025) and -0.050112251237187884 = (60518.29200930938 - 63550.99986292539) / 60518.29200930938 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b5fb7d002acbf729151da9d9918a93c8261d1e95
Compare 225ae771d66dd2732fef7d8ad3a5f2b7b00f8c05
Diff Diff

Regressions in System.Tests.Perf_Decimal

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Mod - Duration of single invocation 23.07 ns 24.48 ns 1.06 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Decimal*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Decimal.Mod

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 24.478156110717418 > 24.15042325474112.
IsChangePoint: Marked as a change because one of 12/6/2022 2:12:43 AM, 1/11/2023 4:46:06 PM, 1/20/2023 8:02:54 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -20.338753347187033 (T) = (0 -24.45885603829905) / Math.Sqrt((0.12477412957692646 / (40)) + (0.06297356546264744 / (22))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (22) - 2, .025) and -0.06873428854401217 = (22.885815773367316 - 24.45885603829905) / 22.885815773367316 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b5fb7d002acbf729151da9d9918a93c8261d1e95
Compare 225ae771d66dd2732fef7d8ad3a5f2b7b00f8c05
Diff Diff

Regressions in System.Diagnostics.Perf_Activity

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
EnumerateActivityLinkTagsSmall - Duration of single invocation 49.34 ns 55.65 ns 1.13 0.13 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Diagnostics.Perf_Activity*'

Payloads

Baseline
Compare

Histogram

System.Diagnostics.Perf_Activity.EnumerateActivityLinkTagsSmall

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 55.651435195133374 > 52.631184336304806.
IsChangePoint: Marked as a change because one of 12/1/2022 8:17:56 AM, 1/20/2023 8:02:54 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -13.206189891761785 (T) = (0 -56.08259984113165) / Math.Sqrt((3.410628006350675 / (41)) + (0.8105108564766561 / (22))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (22) - 2, .025) and -0.08882779704372361 = (51.50731823103848 - 56.08259984113165) / 51.50731823103848 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b5fb7d002acbf729151da9d9918a93c8261d1e95
Compare 225ae771d66dd2732fef7d8ad3a5f2b7b00f8c05
Diff Diff

Regressions in XmlDocumentTests.XmlNodeListTests.Perf_XmlNodeList

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetCount - Duration of single invocation 18.85 ns 20.97 ns 1.11 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'XmlDocumentTests.XmlNodeListTests.Perf_XmlNodeList*'

Payloads

Baseline
Compare

Histogram

XmlDocumentTests.XmlNodeListTests.Perf_XmlNodeList.GetCount

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.974545743075208 > 19.857097631885782.
IsChangePoint: Marked as a change because one of 11/23/2022 1:02:40 AM, 11/24/2022 1:04:57 PM, 12/15/2022 2:06:03 PM, 1/11/2023 4:46:06 PM, 1/20/2023 8:02:54 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -23.202270501911435 (T) = (0 -21.012394715502065) / Math.Sqrt((0.1445498858721582 / (41)) + (0.041353859384398735 / (22))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (22) - 2, .025) and -0.08835619594720877 = (19.30654209876091 - 21.012394715502065) / 19.30654209876091 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Jan 24, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b5fb7d002acbf729151da9d9918a93c8261d1e95
Compare 225ae771d66dd2732fef7d8ad3a5f2b7b00f8c05
Diff Diff

Regressions in IfStatements.IfStatements

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AndArray - Duration of single invocation 78.48 μs 85.33 μs 1.09 0.06 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'IfStatements.IfStatements*'

Payloads

Baseline
Compare

Histogram

IfStatements.IfStatements.AndArray

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 85.32775100019515 > 82.28201700549819.
IsChangePoint: Marked as a change because one of 12/20/2022 9:29:18 AM, 1/20/2023 8:02:54 AM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -22.748111301113273 (T) = (0 -85142.47687561811) / Math.Sqrt((2266756.20828275 / (40)) + (128245.96178257762 / (22))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (22) - 2, .025) and -0.07157398780880879 = (79455.52789100487 - 85142.47687561811) / 79455.52789100487 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@naricc
Copy link

naricc commented Jan 31, 2023

@vargaz Looks like more regressions from this change: dotnet/runtime@bd3e1f5

@naricc naricc removed the untriaged label Jan 31, 2023
@vargaz
Copy link

vargaz commented Jan 31, 2023

That commit is not part of the diff.

@naricc
Copy link

naricc commented Jan 31, 2023

@vargaz The first one I happened to look at had that commit, but looking at it further this issue is kind of funky... It looks like there are actually multiple regressions here at different times.

Some are in this time span around Jan 19: dotnet/runtime@ddb6988...1c8b2de

Some are this:
dotnet/runtime@191b98d...1555786 Which I think we already have issues for. @jandupej is working on it.

But some do have that commit, like look at the static delegate one here: https://pvscmdupload.blob.core.windows.net/autofilereport/autofilereports/01_24_2023/refs/heads/main_x64_ubuntu%2018.04_LLVM=false_MonoAOT=true_MonoInterpreter=false_Regression/PerfLabTests.LowLevelPerf.html

It seems like that is the minority of them though.

@naricc naricc mentioned this issue Jan 31, 2023
Closed
@naricc naricc added the LinkedFromMaster Is this issue linked from the master perf issue label Jan 31, 2023
@naricc
Copy link

naricc commented Feb 1, 2023

@kotlarmilos report this is not showing up in interpreter.

@vargaz
Copy link

vargaz commented Feb 1, 2023

Would be nice to figure out what is happening here, there might be problems with the perf infrastructure, i.e. computing the commit range etc.

@jandupej
Copy link
Member

Remaining regressions after fixing RuntimeHelpers.CreateSpan<T>: dotnet/runtime#83330

@vargaz vargaz removed their assignment Mar 22, 2024
@matouskozak
Copy link
Member

Resolved

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arch-x64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-aot os-linux perf-regression runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@vargaz @naricc @matouskozak @jandupej