Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/arm64: 41 Regressions on 1/15/2023 10:11:50 PM #12073

Closed
performanceautofiler bot opened this issue Jan 24, 2023 · 6 comments
Closed

[Perf] Linux/arm64: 41 Regressions on 1/15/2023 10:11:50 PM #12073

performanceautofiler bot opened this issue Jan 24, 2023 · 6 comments
Assignees
@jandupej
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono mono-aot os-linux perf-regression runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Jan 24, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Basic

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteBasicUtf16 - Duration of single invocation 12.31 ms 19.02 ms 1.54 0.02 False
WriteBasicUtf16 - Duration of single invocation 12.27 ms 18.69 ms 1.52 0.02 False
WriteBasicUtf16 - Duration of single invocation 13.90 ms 20.93 ms 1.51 0.03 False
WriteBasicUtf8 - Duration of single invocation 12.37 ms 18.91 ms 1.53 0.02 False
WriteBasicUtf8 - Duration of single invocation 12.31 ms 18.81 ms 1.53 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Basic*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: True, DataSize: 100000)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.023379329670334 > 12.936974303916669.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -79.60602585258735 (T) = (0 -19286194.71275755) / Math.Sqrt((2670554132.9647303 / (6)) + (227674938159.89743 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.5597887914611581 = (12364619.375608467 - 19286194.71275755) / 12364619.375608467 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: False, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.687541801282052 > 12.882050971865386.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -53.05132702836879 (T) = (0 -19142311.086637024) / Math.Sqrt((7193194383.9596615 / (6)) + (495148507860.8603 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.5572646613081901 = (12292265.767179487 - 19142311.086637024) / 12292265.767179487 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: True, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.930726144444446 > 14.795382020790601.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -73.01550245766249 (T) = (0 -21294329.221174225) / Math.Sqrt((6300389631.05762 / (6)) + (281002878767.7484 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.5151155508178361 = (14054590.892212724 - 21294329.221174225) / 14054590.892212724 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: False, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.908233857988165 > 12.840365774250001.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -82.24514010847278 (T) = (0 -19045396.271168813) / Math.Sqrt((8525182876.9481945 / (6)) + (171977894305.4071 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.5527367146772116 = (12265695.846013429 - 19045396.271168813) / 12265695.846013429 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: True, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.814734620512823 > 12.90625651525.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -59.843761008722545 (T) = (0 -19118190.44609852) / Math.Sqrt((13100129174.435074 / (6)) + (349454030580.9654 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.5584083025847305 = (12267767.320277778 - 19118190.44609852) / 12267767.320277778 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Perf_Convert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ChangeType - Duration of single invocation 113.59 ns 191.29 ns 1.68 0.22 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Perf_Convert*'

Payloads

Baseline
Compare

Histogram

System.Perf_Convert.ChangeType

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 191.2921732135211 > 119.25171116160135.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -78.17671240749297 (T) = (0 -186.5749840331377) / Math.Sqrt((0.23611605741144293 / (6)) + (26.376056480773325 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.6376984499439768 = (113.92511487051853 - 186.5749840331377) / 113.92511487051853 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Tests.Perf_Enum

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
StringFormat - Duration of single invocation 1.15 μs 1.42 μs 1.24 0.04 False
InterpolateIntoSpan_Flags - Duration of single invocation 1.38 μs 1.61 μs 1.17 0.21 False
ToString_Flags - Duration of single invocation 162.57 ns 227.00 ns 1.40 0.15 False
InterpolateIntoSpan_NonFlags - Duration of single invocation 1.42 μs 1.86 μs 1.30 0.19 False
InterpolateIntoString - Duration of single invocation 1.27 μs 1.39 μs 1.09 0.21 False
ToString_Flags - Duration of single invocation 158.85 ns 247.67 ns 1.56 0.19 False
InterpolateIntoString - Duration of single invocation 1.59 μs 1.74 μs 1.09 0.12 False
InterpolateIntoString - Duration of single invocation 1.45 μs 1.80 μs 1.24 0.23 False
InterpolateIntoSpan_Flags - Duration of single invocation 1.14 μs 1.37 μs 1.20 0.19 False
InterpolateIntoStringBuilder_NonFlags - Duration of single invocation 1.45 μs 1.72 μs 1.19 0.17 False
StringFormat - Duration of single invocation 926.25 ns 999.33 ns 1.08 0.02 False
ToString_NonFlags_Small - Duration of single invocation 145.63 ns 226.17 ns 1.55 0.29 False
InterpolateIntoSpan_Flags - Duration of single invocation 1.43 μs 1.73 μs 1.22 0.40 False
ToString_NonFlags_Large - Duration of single invocation 112.34 ns 191.82 ns 1.71 0.21 False
StringFormat - Duration of single invocation 1.24 μs 1.36 μs 1.10 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Enum*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Enum.StringFormat(value: 32)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4243960761071093 > 1.2211591373608768.
IsChangePoint: Marked as a change because one of 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -38.396696453913584 (T) = (0 -1440.9381922420584) / Math.Sqrt((224.208654753078 / (6)) + (559.2618200678658 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.24585898144433174 = (1156.5820961306313 - 1440.9381922420584) / 1156.5820961306313 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.InterpolateIntoSpan_Flags(value: Red, Green)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.6095103074870853 > 1.470660970621768.
IsChangePoint: Marked as a change because one of 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -8.91075698987932 (T) = (0 -1534.1205222104966) / Math.Sqrt((783.5649189697091 / (6)) + (2259.0999680598675 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.08978420316417078 = (1407.7287207469171 - 1534.1205222104966) / 1407.7287207469171 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.ToString_Flags(value: 36)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 227.00190449564954 > 170.21553777752885.
IsChangePoint: Marked as a change because one of 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -54.66920495044473 (T) = (0 -230.5147401608301) / Math.Sqrt((2.745208778926964 / (6)) + (35.36350221221346 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.4213943374759428 = (162.1750798375694 - 230.5147401608301) / 162.1750798375694 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.InterpolateIntoSpan_NonFlags(value: 42)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.85632427223479 > 1.4790402739170492.
IsChangePoint: Marked as a change because one of 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -18.13949794155028 (T) = (0 -1715.6863817028168) / Math.Sqrt((915.9710878820558 / (6)) + (4987.531614065618 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.22806067194231194 = (1397.0697221247804 - 1715.6863817028168) / 1397.0697221247804 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.InterpolateIntoString(value: Red)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.394226678876679 > 1.3436510152655308.
IsChangePoint: Marked as a change because one of 1/16/2023 10:36:41 AM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -9.180430259345806 (T) = (0 -1402.2903665545205) / Math.Sqrt((680.6536772236541 / (9)) + (1543.388954695735 / (29))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (9) + (29) - 2, .025) and -0.08027874844613055 = (1298.081970576178 - 1402.2903665545205) / 1298.081970576178 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.ToString_Flags(value: 32)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 247.6682947393601 > 166.37092818235553.
IsChangePoint: Marked as a change because one of 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -40.913807928724125 (T) = (0 -226.97038968455513) / Math.Sqrt((4.93709611517003 / (6)) + (62.02892583449347 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.42763185826496886 = (158.9838363234602 - 226.97038968455513) / 158.9838363234602 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.InterpolateIntoString(value: Red, Green)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7434020028331239 > 1.6866885209053128.
IsChangePoint: Marked as a change because one of 1/16/2023 10:36:41 AM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -7.2845883183889395 (T) = (0 -1745.4241593828674) / Math.Sqrt((1573.3256038041507 / (9)) + (2019.6905652613257 / (29))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (9) + (29) - 2, .025) and -0.06980923981415756 = (1631.528401910302 - 1745.4241593828674) / 1631.528401910302 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.InterpolateIntoString(value: 32)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.8012072000077495 > 1.5853500621531613.
IsChangePoint: Marked as a change because one of 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -20.97560289223963 (T) = (0 -1817.1675404361097) / Math.Sqrt((875.8917809911931 / (6)) + (2725.963644244449 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.21286015616643206 = (1498.2498445490635 - 1817.1675404361097) / 1498.2498445490635 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.InterpolateIntoSpan_Flags(value: Red)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3675668599731556 > 1.2316896946323168.
IsChangePoint: Marked as a change because one of 1/15/2023 10:11:50 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -7.291831336888016 (T) = (0 -1280.2118873442416) / Math.Sqrt((1181.0829521274552 / (7)) + (1533.9599685623868 / (31))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (31) - 2, .025) and -0.09186723367108132 = (1172.497761508885 - 1280.2118873442416) / 1172.497761508885 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.InterpolateIntoStringBuilder_NonFlags(value: 42)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7186138289878685 > 1.4699062257810949.
IsChangePoint: Marked as a change because one of 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -17.366276449130922 (T) = (0 -1745.614751689926) / Math.Sqrt((1821.9264173942342 / (6)) + (2191.567100015199 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.23749566044723275 = (1410.602725717081 - 1745.614751689926) / 1410.602725717081 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.StringFormat(value: Red)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 999.3338136915448 > 964.9236890537061.
IsChangePoint: Marked as a change because one of 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -20.666687712574024 (T) = (0 -1010.3936383312808) / Math.Sqrt((85.83149107227042 / (6)) + (133.24094672015266 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.09637427688535238 = (921.5772931135062 - 1010.3936383312808) / 921.5772931135062 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.ToString_NonFlags_Small(value: -1)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 226.17187377031442 > 153.6539322601954.
IsChangePoint: Marked as a change because one of 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -35.95203329529247 (T) = (0 -206.47101803672538) / Math.Sqrt((1.0454323300505037 / (6)) + (82.84918692948186 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.4073643527887726 = (146.7075797589951 - 206.47101803672538) / 146.7075797589951 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.InterpolateIntoSpan_Flags(value: 32)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7336627145293677 > 1.4942832156275154.
IsChangePoint: Marked as a change because one of 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -13.765711685900598 (T) = (0 -1696.0904173448348) / Math.Sqrt((2196.2047963820505 / (6)) + (3595.373472138343 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.21583094086050558 = (1395.0051445017718 - 1696.0904173448348) / 1395.0051445017718 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.ToString_NonFlags_Large(value: 42)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 191.819350582269 > 119.91017708189437.
IsChangePoint: Marked as a change because one of 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -57.53443800173162 (T) = (0 -186.38381660301724) / Math.Sqrt((4.885196252778727 / (6)) + (25.073839842598797 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.6398523469679864 = (113.6589016368654 - 186.38381660301724) / 113.6589016368654 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.StringFormat(value: Red, Green)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3553437035985678 > 1.3029629395096864.
IsChangePoint: Marked as a change because one of 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -17.168857107839674 (T) = (0 -1339.5798732144724) / Math.Sqrt((168.32702117205062 / (6)) + (274.13018014408516 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.08408145366870698 = (1235.6819394715383 - 1339.5798732144724) / 1235.6819394715383 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Tests.Perf_UInt32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 74.36 ns 134.60 ns 1.81 0.22 False
TryFormat - Duration of single invocation 45.30 ns 106.20 ns 2.34 0.11 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt32*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt32.ToString(value: 12345)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 134.59933088404946 > 78.38559114899806.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -76.02838615251466 (T) = (0 -135.17300423990173) / Math.Sqrt((0.0653657450669726 / (6)) + (19.979781530865655 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.8125578132319438 = (74.5758304938571 - 135.17300423990173) / 74.5758304938571 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt32.TryFormat(value: 0)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 106.19983865973451 > 51.55274297126059.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -37.992760468006736 (T) = (0 -105.01095372321363) / Math.Sqrt((6.818779762080176 / (6)) + (37.050573399131494 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -1.2124523535503229 = (47.46360008824711 - 105.01095372321363) / 47.46360008824711 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Xml.Tests.Perf_XmlConvert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TimeSpan_ToString - Duration of single invocation 490.32 ns 750.33 ns 1.53 0.35 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Xml.Tests.Perf_XmlConvert*'

Payloads

Baseline
Compare

Histogram

System.Xml.Tests.Perf_XmlConvert.TimeSpan_ToString

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 750.3314249379252 > 515.9748910303955.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -84.19643727854209 (T) = (0 -726.321380073139) / Math.Sqrt((21.472049730935634 / (6)) + (135.9639387898568 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.47999711391835087 = (490.758646245042 - 726.321380073139) / 490.758646245042 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Tests.Perf_Uri

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
UriBuilderReplacePort - Duration of single invocation 696.77 ns 766.70 ns 1.10 0.06 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Uri*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Uri.UriBuilderReplacePort

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 766.7012369524747 > 731.9815451922999.
IsChangePoint: Marked as a change because one of 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -19.50076925640439 (T) = (0 -768.1463254274661) / Math.Sqrt((42.374277721589934 / (6)) + (154.42102705783086 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.09592787605207709 = (700.909560029263 - 768.1463254274661) / 700.909560029263 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Jan 24, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Tests.Perf_Int64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryFormat - Duration of single invocation 104.87 ns 173.39 ns 1.65 0.05 False
ToString - Duration of single invocation 85.93 ns 147.71 ns 1.72 0.25 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int64.TryFormat(value: 12345)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 173.3864081464679 > 109.45302017962709.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -48.533649016192584 (T) = (0 -176.93297674646945) / Math.Sqrt((7.838560137463384 / (6)) + (26.831896069370426 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.6715010922602986 = (105.85274371984444 - 176.93297674646945) / 105.85274371984444 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int64.ToString(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 147.7071566132008 > 90.27987662303471.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -62.566045353401364 (T) = (0 -158.56950352634883) / Math.Sqrt((0.14032307979027794 / (6)) + (42.55291173064257 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.8483632588523752 = (85.7891449458926 - 158.56950352634883) / 85.7891449458926 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Tests.Perf_Version

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString3 - Duration of single invocation 216.79 ns 406.60 ns 1.88 0.04 False
ToString2 - Duration of single invocation 173.15 ns 297.14 ns 1.72 0.09 False
ToString4 - Duration of single invocation 279.12 ns 512.66 ns 1.84 0.07 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Version*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Version.ToString3

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 406.59942357636714 > 231.83464583300554.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -77.39500030953282 (T) = (0 -410.1571486547461) / Math.Sqrt((10.01811071804953 / (6)) + (137.125286405163 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.8534530513972568 = (221.29351932898555 - 410.1571486547461) / 221.29351932898555 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.ToString2

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 297.14052556077206 > 182.63044901852692.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -84.41162145551091 (T) = (0 -292.1100258000584) / Math.Sqrt((6.70243337675732 / (6)) + (28.06561031077654 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.6893809943785508 = (172.90950162933072 - 292.1100258000584) / 172.90950162933072 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.ToString4

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 512.6565935792579 > 288.5134988755248.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -104.32858887485564 (T) = (0 -521.78829805139) / Math.Sqrt((5.839405308004738 / (6)) + (148.47313093177277 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.9000723246615656 = (274.61496664045626 - 521.78829805139) / 274.61496664045626 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in PerfLabTests.LowLevelPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IntegerFormatting - Duration of single invocation 11.78 ms 17.51 ms 1.49 0.09 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.LowLevelPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.LowLevelPerf.IntegerFormatting

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.505756557142856 > 12.383798969062502.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -64.89759090772776 (T) = (0 -17951887.915712487) / Math.Sqrt((2292307011.5223627 / (6)) + (275115728108.14575 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.5210620885184366 = (11802205.873922085 - 17951887.915712487) / 11802205.873922085 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Buffers.Text.Tests.Utf8FormatterTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FormatterInt64 - Duration of single invocation 97.04 ns 145.92 ns 1.50 0.34 False
FormatterInt32 - Duration of single invocation 96.39 ns 153.55 ns 1.59 0.37 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: 12345)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 145.92314638623802 > 102.21361562863542.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -92.24424697921735 (T) = (0 -151.83767507242396) / Math.Sqrt((0.28802793215713274 / (6)) + (9.508660666219885 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.5550025500813122 = (97.6446469907617 - 151.83767507242396) / 97.6446469907617 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 153.54987094079883 > 100.29946889893732.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -55.331723567976056 (T) = (0 -150.4999239124787) / Math.Sqrt((3.1130807974686983 / (6)) + (14.353095043217552 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.5664329509457401 = (96.07811417757381 - 150.4999239124787) / 96.07811417757381 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Tests.Perf_Int32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 77.87 ns 144.07 ns 1.85 0.15 False
TryFormat - Duration of single invocation 47.93 ns 124.87 ns 2.61 0.09 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int32*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int32.ToString(value: 12345)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 144.07463314129126 > 80.88347353063963.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -74.713534191496 (T) = (0 -139.61055007679568) / Math.Sqrt((1.319249601293165 / (6)) + (15.831283262988256 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.8261182464444979 = (76.45208646735841 - 139.61055007679568) / 76.45208646735841 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int32.TryFormat(value: 4)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 124.87460900522697 > 51.017526922320826.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -47.5026317803588 (T) = (0 -105.322841329737) / Math.Sqrt((0.1798692402393473 / (6)) + (44.74346521744309 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -1.1692125901180674 = (48.553489782209134 - 105.322841329737) / 48.553489782209134 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Tests.Perf_Byte

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 61.20 ns 120.81 ns 1.97 0.23 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Byte*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Byte.ToString(value: 255)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 120.81218588292585 > 64.05294838252719.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -54.47895457346552 (T) = (0 -123.9543816149178) / Math.Sqrt((0.36044032046085156 / (6)) + (41.18506963721081 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -1.0412944470722927 = (60.72342076504358 - 123.9543816149178) / 60.72342076504358 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Tests.Perf_UInt64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 83.93 ns 162.47 ns 1.94 0.23 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt64.ToString(value: 12345)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 162.4654215266458 > 87.62802005588377.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -111.3272286074471 (T) = (0 -161.34566169336458) / Math.Sqrt((0.486260503229457 / (6)) + (13.09350926533665 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.9346127721951212 = (83.39946061158929 - 161.34566169336458) / 83.39946061158929 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Numerics.Tests.Perf_BigInteger

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToStringD - Duration of single invocation 162.92 ns 241.06 ns 1.48 0.17 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BigInteger*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_BigInteger.ToStringD(numberString: 123)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 241.0609387310077 > 174.147302540669.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -49.995311365440244 (T) = (0 -231.99376174439723) / Math.Sqrt((6.25861220358184 / (6)) + (26.0256029127423 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.41567317919088154 = (163.87522569085593 - 231.99376174439723) / 163.87522569085593 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Tests.Perf_UInt16

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 74.38 ns 134.60 ns 1.81 0.19 False
ToString - Duration of single invocation 74.42 ns 133.01 ns 1.79 0.23 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt16*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt16.ToString(value: 12345)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 134.60447155551228 > 78.2624204195619.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -85.42760627876675 (T) = (0 -134.30917013385508) / Math.Sqrt((1.1399747564864042 / (6)) + (9.487842362787658 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.7973947967321299 = (74.72435681801493 - 134.30917013385508) / 74.72435681801493 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt16.ToString(value: 65535)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 133.01310976686685 > 78.21393291621156.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -58.86268894464459 (T) = (0 -135.66114906713136) / Math.Sqrt((0.026925769394334637 / (6)) + (34.42627531153018 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.8214316357570316 = (74.48050555614033 - 135.66114906713136) / 74.48050555614033 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Tests.Perf_Int16

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 76.20 ns 135.93 ns 1.78 0.17 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int16*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int16.ToString(value: 32767)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 135.9268169919976 > 79.89390237914398.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/24/2023 3:05:55 AM falls between 1/15/2023 7:15:22 AM and 1/24/2023 3:05:55 AM.
IsRegressionStdDev: Marked as regression because -66.71073243955428 (T) = (0 -139.55627727733892) / Math.Sqrt((0.08830105164159045 / (6)) + (28.465522614852727 / (32))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (32) - 2, .025) and -0.833392211675469 = (76.11916118581284 - 139.55627727733892) / 76.11916118581284 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@naricc
Copy link

naricc commented Jan 25, 2023

It looks like one of these based on the time, but I can't see how any of these changes could have caused this regression?
dotnet/runtime@e2e690b...ecc3a7b

@naricc naricc removed the untriaged label Jan 25, 2023
@lewing
Copy link
Member

lewing commented Jan 29, 2023

it is dotnet/runtime@674514a again

@naricc
Copy link

naricc commented Jan 30, 2023

@jandupej You are already working on a related issue, right?

@jandupej
Copy link
Member

jandupej commented Jan 30, 2023 •
edited
Loading

@naricc I am working on #80762. It could help with the regressions here.

@naricc naricc mentioned this issue Jan 31, 2023
Closed
@kotlarmilos
Copy link
Member

Mitigated by dotnet/runtime#81695

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jandupej jandupej

Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono mono-aot os-linux perf-regression runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@lewing @kotlarmilos @naricc @jandupej