Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 27 Regressions on 8/17/2022 6:09:21 PM #74395

Closed
performanceautofiler bot opened this issue Aug 20, 2022 · 21 comments
Closed

[Perf] Linux/x64: 27 Regressions on 8/17/2022 6:09:21 PM #74395

performanceautofiler bot opened this issue Aug 20, 2022 · 21 comments
Assignees
@radekdoulik @adamsitnik
Labels
arch-wasm WebAssembly architecture area-Codegen-AOT-mono blocking-release
Milestone
7.0.0

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Aug 20, 2022 •
edited by naricc
Loading

There are some large wasm regressions here; as much as a 10 x slow down. Unfortunately, there seems to be some kind of glitch in the reporting, and the narrowest I can get the change range is this: 543bcc5...a250bfe.

@lewing @radical @pavelsavara Any likely candidates in that date range?

Automated issue reporting below.

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Uri_IsNotMatch - Duration of single invocation 444.95 ns 592.31 ns 1.33 0.06 False
Uri_IsMatch - Duration of single invocation 455.72 ns 505.99 ns 1.11 0.05 False
Backtracking - Duration of single invocation 2.45 μs 2.93 μs 1.19 0.10 False
Backtracking - Duration of single invocation 2.45 μs 2.91 μs 1.19 0.10 False
Uri_IsNotMatch - Duration of single invocation 446.25 ns 581.71 ns 1.30 0.05 False
Uri_IsNotMatch - Duration of single invocation 447.94 ns 593.46 ns 1.32 0.04 False

graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: None)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 592.3093690327008 > 465.23264235782176.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -39.47621223602034 (T) = (0 -594.5364016454982) / Math.Sqrt((78.92038015570309 / (14)) + (73.27831635677934 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.32709221691484786 = (447.99931313563445 - 594.5364016454982) / 447.99931313563445 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsMatch(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 505.9861520887883 > 482.7014638466213.
IsChangePoint: Marked as a change because one of 6/24/2022 10:40:04 AM, 6/25/2022 6:06:34 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -14.329767862712716 (T) = (0 -514.4415933301957) / Math.Sqrt((16.440001607480827 / (14)) + (123.32550738351799 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.12037177742305741 = (459.17043225905917 - 514.4415933301957) / 459.17043225905917 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.9266354356553093 > 2.591927353654952.
IsChangePoint: Marked as a change because one of 6/24/2022 6:48:53 PM, 6/29/2022 4:06:39 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -50.14154390540031 (T) = (0 -2934.2504087143566) / Math.Sqrt((556.432351718883 / (14)) + (417.26555557386655 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.18845364681210353 = (2468.964958444245 - 2934.2504087143566) / 2468.964958444245 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.9101414399726724 > 2.5780127447204717.
IsChangePoint: Marked as a change because one of 6/24/2022 5:01:29 PM, 6/25/2022 2:40:01 AM, 6/29/2022 4:06:39 AM, 7/12/2022 12:16:14 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -61.70631510275718 (T) = (0 -2923.812224534253) / Math.Sqrt((407.1945204507192 / (14)) + (236.0815117136678 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.18619294374645873 = (2464.870694053968 - 2923.812224534253) / 2464.870694053968 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 581.7136123036038 > 471.5091607783785.
IsChangePoint: Marked as a change because one of 6/25/2022 5:15:47 AM, 6/26/2022 6:11:06 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -65.94725454140149 (T) = (0 -592.9345145576187) / Math.Sqrt((24.613985181656737 / (14)) + (26.562821064295214 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.3181628970020087 = (449.81884705310074 - 592.9345145576187) / 449.81884705310074 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 593.4594763018782 > 471.43001759852376.
IsChangePoint: Marked as a change because one of 6/24/2022 10:40:04 AM, 6/24/2022 6:48:53 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -48.543170908408 (T) = (0 -596.0567739968475) / Math.Sqrt((34.7262092388468 / (14)) + (56.62787404726851 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.3178926565327463 = (452.2802149645615 - 596.0567739968475) / 452.2802149645615 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Reader

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadSpanEmptyLoop - Duration of single invocation 92.65 μs 99.48 μs 1.07 0.09 False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 1.70 μs 1.91 μs 1.13 0.07 False
ReadSpanEmptyLoop - Duration of single invocation 4.25 μs 4.95 μs 1.16 0.09 False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 11.52 μs 12.59 μs 1.09 0.12 False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 112.87 μs 118.86 μs 1.05 0.07 False

graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Reader*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: True, TestCase: Json40KB)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 99.47909983507735 > 97.49529907633277.
IsChangePoint: Marked as a change because one of 6/17/2022 9:56:33 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -9.875672080579108 (T) = (0 -101229.28427961336) / Math.Sqrt((265137.5448871564 / (14)) + (5976092.35420695 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.08769210576340399 = (93067.95897775218 - 101229.28427961336) / 93067.95897775218 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json400B)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.90914647431983 > 1.777974693072054.
IsChangePoint: Marked as a change because one of 6/22/2022 10:31:12 AM, 7/19/2022 8:53:09 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -24.12292592652499 (T) = (0 -1927.1512790997767) / Math.Sqrt((611.4742780301284 / (14)) + (471.64347753333664 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.1398571366423715 = (1690.6954539728583 - 1927.1512790997767) / 1690.6954539728583 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: False, TestCase: LotsOfNumbers)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.946905717995979 > 4.460535892543025.
IsChangePoint: Marked as a change because one of 6/24/2022 6:48:53 PM, 6/25/2022 5:15:47 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -14.376124247128779 (T) = (0 -4757.429574447925) / Math.Sqrt((4758.814727171611 / (14)) + (8231.09975483114 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.11985725605871154 = (4248.246415968665 - 4757.429574447925) / 4248.246415968665 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json4KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.58755040658602 > 12.38535184349092.
IsChangePoint: Marked as a change because one of 6/22/2022 10:31:12 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -6.466117999484596 (T) = (0 -12591.240550799725) / Math.Sqrt((69596.82198162597 / (14)) + (79394.98687585884 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.06418279234267017 = (11831.840019778581 - 12591.240550799725) / 11831.840019778581 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json40KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 118.8626892992424 > 118.55053428507193.
IsChangePoint: Marked as a change because one of 6/17/2022 9:56:33 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -7.506023118864004 (T) = (0 -121572.12886191734) / Math.Sqrt((4192742.5510894423 / (14)) + (6482122.752315378 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.06649265512776761 = (113992.4670623754 - 121572.12886191734) / 113992.4670623754 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Format_MultipleArgs - Duration of single invocation 529.02 ns 850.06 ns 1.61 0.14 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.Format_MultipleArgs

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 850.0557408302284 > 547.6817332098481.
IsChangePoint: Marked as a change because one of 6/24/2022 10:40:04 AM, 6/27/2022 8:54:23 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -86.40554379355675 (T) = (0 -848.1242802325862) / Math.Sqrt((88.87602574889881 / (14)) + (69.21223243326581 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.6173858106947179 = (524.3796963127123 - 848.1242802325862) / 524.3796963127123 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Tests.Perf_Environment

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ExpandEnvironmentVariables - Duration of single invocation 382.06 ns 1.06 μs 2.76 0.11 False
SetEnvironmentVariable - Duration of single invocation 194.79 ns 661.73 ns 3.40 0.16 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Environment*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Environment.ExpandEnvironmentVariables

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.05539799053797 > 404.12774324999464.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -100.66329094480135 (T) = (0 -1058.1419394441955) / Math.Sqrt((68.03089949492843 / (14)) + (358.51775965172686 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -1.7472293516620847 = (385.16694603747425 - 1058.1419394441955) / 385.16694603747425 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Environment.SetEnvironmentVariable

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 661.725814258165 > 213.09369271099766.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -164.42300266168527 (T) = (0 -647.4959303229203) / Math.Sqrt((49.2398659816711 / (14)) + (35.11775717258779 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -2.2433183359130027 = (199.6399561379005 - 647.4959303229203) / 199.6399561379005 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Memory.ReadOnlySpan

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfString - Duration of single invocation 24.67 ns 36.76 ns 1.49 0.21 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.ReadOnlySpan*'

Payloads

Baseline
Compare

Histogram

System.Memory.ReadOnlySpan.IndexOfString(input: "StrIng", value: "string", comparisonType: OrdinalIgnoreCase)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.76482912607041 > 26.19735640294146.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -67.5375793028699 (T) = (0 -36.38052307472043) / Math.Sqrt((0.037995025208353364 / (14)) + (0.23599627187678154 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.4615325383200185 = (24.892037721266604 - 36.38052307472043) / 24.892037721266604 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Collections.ContainsFalse<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ICollection - Duration of single invocation 186.65 μs 2.42 ms 12.95 0.01 False
Stack - Duration of single invocation 148.17 μs 2.98 ms 20.11 0.01 False
Queue - Duration of single invocation 186.46 μs 2.42 ms 12.98 0.02 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsFalse&lt;Int32&gt;*'
### Histogram

System.Collections.ContainsFalse<Int32>.ICollection(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.4180025503826528 > 196.43006513020836.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -373.14299335258266 (T) = (0 -2427206.700892857) / Math.Sqrt((1083311.6941798208 / (14)) + (323590191.29539 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -11.95486324505923 = (187358.7281454749 - 2427206.700892857) / 187358.7281454749 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsFalse&lt;Int32&gt;.Stack(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.9792487988095235 > 155.34581797022406.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -42.64814865820476 (T) = (0 -2532821.408927845) / Math.Sqrt((874624.8610895702 / (14)) + (28136948852.7768 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -16.092038104366907 = (148187.20818792962 - 2532821.408927845) / 148187.20818792962 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsFalse&lt;Int32&gt;.Queue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.4195034313186814 > 195.68580234160373.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -91.92109792877557 (T) = (0 -2470159.070670352) / Math.Sqrt((714329.7028732462 / (14)) + (5552447103.319803 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -12.216076602459486 = (186905.62600179392 - 2470159.070670352) / 186905.62600179392 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Load - Duration of single invocation 277.65 μs 294.38 μs 1.06 0.09 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks*'

Payloads

Baseline
Compare

Histogram

Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks.Load(FileName: "repeated.xml")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 294.3815035812672 > 286.28722408634746.
IsChangePoint: Marked as a change because one of 7/1/2022 12:02:05 PM, 7/10/2022 5:59:30 PM, 7/18/2022 3:18:54 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -9.519909182959154 (T) = (0 -300647.38905054185) / Math.Sqrt((42979330.934151694 / (14)) + (35851042.04585498 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.09181732773423353 = (275364.18539395486 - 300647.38905054185) / 275364.18539395486 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Segment

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadSingleSegmentSequenceByN - Duration of single invocation 117.72 μs 124.68 μs 1.06 0.05 False
ReadSingleSegmentSequenceByN - Duration of single invocation 11.91 μs 12.64 μs 1.06 0.04 False
ReadSingleSegmentSequenceByN - Duration of single invocation 11.86 μs 13.09 μs 1.10 0.04 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Segment*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Segment.ReadSingleSegmentSequenceByN(numberOfBytes: 8192, TestCase: Json40KB)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 124.68098554894179 > 124.27589552531899.
IsChangePoint: Marked as a change because one of 6/11/2022 8:34:59 AM, 7/19/2022 2:55:40 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -5.581969937441441 (T) = (0 -127368.57600209878) / Math.Sqrt((7023383.564901864 / (14)) + (15611587.1916289 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.07013381276822936 = (119021.16771043884 - 127368.57600209878) / 119021.16771043884 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Segment.ReadSingleSegmentSequenceByN(numberOfBytes: 4096, TestCase: Json4KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.640910305240409 > 12.515637436028303.
IsChangePoint: Marked as a change because one of 6/17/2022 9:56:33 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -10.206712835881275 (T) = (0 -12774.988873320122) / Math.Sqrt((20535.81036398369 / (14)) + (46387.504846041105 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.06953141358175953 = (11944.472795369229 - 12774.988873320122) / 11944.472795369229 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Segment.ReadSingleSegmentSequenceByN(numberOfBytes: 8192, TestCase: Json4KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.090807536867004 > 12.469607141755468.
IsChangePoint: Marked as a change because one of 6/17/2022 9:56:33 AM, 7/19/2022 8:53:09 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -12.446003189221445 (T) = (0 -12721.135652115448) / Math.Sqrt((3972.5173419869543 / (14)) + (40342.44528186691 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.07243763434320451 = (11861.888509634673 - 12721.135652115448) / 11861.888509634673 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Memory.Span<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfValue - Duration of single invocation 138.67 ns 2.42 μs 17.43 0.01 False
IndexOfAnyTwoValues - Duration of single invocation 200.08 ns 3.04 μs 15.18 0.01 False
IndexOfAnyThreeValues - Duration of single invocation 276.67 ns 3.75 μs 13.54 0.01 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.Span&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Byte>.IndexOfValue(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.4170771770847206 > 146.37516945438924.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -14.608891603462073 (T) = (0 -2611.6152103241934) / Math.Sqrt((259.65556211673896 / (14)) + (254663.71967576197 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -16.02608588753459 = (153.38905415931507 - 2611.6152103241934) / 153.38905415931507 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.IndexOfAnyTwoValues(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.0370345882066276 > 209.74119752252065.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -70.32494865522497 (T) = (0 -3138.38731756492) / Math.Sqrt((0.1937329534100386 / (14)) + (15711.882878990908 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -14.689264738200135 = (200.0340595900324 - 3138.38731756492) / 200.0340595900324 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.IndexOfAnyThreeValues(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.7468551530143617 > 290.8288738091121.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -23.271230281894688 (T) = (0 -3929.8603884016356) / Math.Sqrt((1.2682850554058953 / (14)) + (221737.7219606583 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -13.181022917520329 = (277.12108014058583 - 3929.8603884016356) / 277.12108014058583 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.IO.Tests.Perf_Path

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetFileNameWithoutExtension - Duration of single invocation 109.03 ns 153.38 ns 1.41 0.25 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_Path*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_Path.GetFileNameWithoutExtension

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 153.38430839699873 > 114.74546136153367.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -27.261989122675345 (T) = (0 -151.75101640427368) / Math.Sqrt((12.744288495830018 / (14)) + (13.286249350442171 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.384142762437261 = (109.63537903927168 - 151.75101640427368) / 109.63537903927168 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Collections.ContainsTrue<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableArray - Duration of single invocation 125.83 μs 1.25 ms 9.90 0.07 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsTrue&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsTrue<Int32>.ImmutableArray(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.245679729474852 > 132.27599546370968.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -100.87925648694225 (T) = (0 -1265867.2470181838) / Math.Sqrt((1331132.729081719 / (14)) + (1147780671.5517871 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -9.02935874958368 = (126216.16980953344 - 1265867.2470181838) / 126216.16980953344 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler performanceautofiler bot added AOT=true PGO untriaged New issue has not been triaged by the area owner labels Aug 20, 2022
@naricc
Copy link
Contributor

naricc commented Aug 23, 2022

Something in this date range: 543bcc5...a250bfe

@naricc naricc transferred this issue from dotnet/perf-autofiling-issues Aug 23, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@naricc naricc self-assigned this Aug 23, 2022
@naricc naricc added area-Codegen-AOT-mono arch-wasm WebAssembly architecture labels Aug 23, 2022
@ghost
Copy link

ghost commented Aug 23, 2022

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

There are some large wasm regressions here; as much as a 10 x slow down. Unfortunately, there seems to be some kind of glitch in the reporting, and the narrowest I can get the change range is this: 543bcc5...a250bfe.

@lewing @radical @pavelsavara Any likely candidates in that date range?

Automated issue reporting below.

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Uri_IsNotMatch - Duration of single invocation 444.95 ns 592.31 ns 1.33 0.06 False
Uri_IsMatch - Duration of single invocation 455.72 ns 505.99 ns 1.11 0.05 False
Backtracking - Duration of single invocation 2.45 μs 2.93 μs 1.19 0.10 False
Backtracking - Duration of single invocation 2.45 μs 2.91 μs 1.19 0.10 False
Uri_IsNotMatch - Duration of single invocation 446.25 ns 581.71 ns 1.30 0.05 False
Uri_IsNotMatch - Duration of single invocation 447.94 ns 593.46 ns 1.32 0.04 False

graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: None)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 592.3093690327008 > 465.23264235782176.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -39.47621223602034 (T) = (0 -594.5364016454982) / Math.Sqrt((78.92038015570309 / (14)) + (73.27831635677934 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.32709221691484786 = (447.99931313563445 - 594.5364016454982) / 447.99931313563445 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsMatch(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 505.9861520887883 > 482.7014638466213.
IsChangePoint: Marked as a change because one of 6/24/2022 10:40:04 AM, 6/25/2022 6:06:34 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -14.329767862712716 (T) = (0 -514.4415933301957) / Math.Sqrt((16.440001607480827 / (14)) + (123.32550738351799 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.12037177742305741 = (459.17043225905917 - 514.4415933301957) / 459.17043225905917 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.9266354356553093 > 2.591927353654952.
IsChangePoint: Marked as a change because one of 6/24/2022 6:48:53 PM, 6/29/2022 4:06:39 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -50.14154390540031 (T) = (0 -2934.2504087143566) / Math.Sqrt((556.432351718883 / (14)) + (417.26555557386655 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.18845364681210353 = (2468.964958444245 - 2934.2504087143566) / 2468.964958444245 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.9101414399726724 > 2.5780127447204717.
IsChangePoint: Marked as a change because one of 6/24/2022 5:01:29 PM, 6/25/2022 2:40:01 AM, 6/29/2022 4:06:39 AM, 7/12/2022 12:16:14 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -61.70631510275718 (T) = (0 -2923.812224534253) / Math.Sqrt((407.1945204507192 / (14)) + (236.0815117136678 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.18619294374645873 = (2464.870694053968 - 2923.812224534253) / 2464.870694053968 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 581.7136123036038 > 471.5091607783785.
IsChangePoint: Marked as a change because one of 6/25/2022 5:15:47 AM, 6/26/2022 6:11:06 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -65.94725454140149 (T) = (0 -592.9345145576187) / Math.Sqrt((24.613985181656737 / (14)) + (26.562821064295214 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.3181628970020087 = (449.81884705310074 - 592.9345145576187) / 449.81884705310074 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 593.4594763018782 > 471.43001759852376.
IsChangePoint: Marked as a change because one of 6/24/2022 10:40:04 AM, 6/24/2022 6:48:53 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -48.543170908408 (T) = (0 -596.0567739968475) / Math.Sqrt((34.7262092388468 / (14)) + (56.62787404726851 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.3178926565327463 = (452.2802149645615 - 596.0567739968475) / 452.2802149645615 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Reader

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadSpanEmptyLoop - Duration of single invocation 92.65 μs 99.48 μs 1.07 0.09 False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 1.70 μs 1.91 μs 1.13 0.07 False
ReadSpanEmptyLoop - Duration of single invocation 4.25 μs 4.95 μs 1.16 0.09 False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 11.52 μs 12.59 μs 1.09 0.12 False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 112.87 μs 118.86 μs 1.05 0.07 False

graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Reader*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: True, TestCase: Json40KB)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 99.47909983507735 > 97.49529907633277.
IsChangePoint: Marked as a change because one of 6/17/2022 9:56:33 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -9.875672080579108 (T) = (0 -101229.28427961336) / Math.Sqrt((265137.5448871564 / (14)) + (5976092.35420695 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.08769210576340399 = (93067.95897775218 - 101229.28427961336) / 93067.95897775218 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json400B)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.90914647431983 > 1.777974693072054.
IsChangePoint: Marked as a change because one of 6/22/2022 10:31:12 AM, 7/19/2022 8:53:09 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -24.12292592652499 (T) = (0 -1927.1512790997767) / Math.Sqrt((611.4742780301284 / (14)) + (471.64347753333664 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.1398571366423715 = (1690.6954539728583 - 1927.1512790997767) / 1690.6954539728583 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: False, TestCase: LotsOfNumbers)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.946905717995979 > 4.460535892543025.
IsChangePoint: Marked as a change because one of 6/24/2022 6:48:53 PM, 6/25/2022 5:15:47 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -14.376124247128779 (T) = (0 -4757.429574447925) / Math.Sqrt((4758.814727171611 / (14)) + (8231.09975483114 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.11985725605871154 = (4248.246415968665 - 4757.429574447925) / 4248.246415968665 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json4KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.58755040658602 > 12.38535184349092.
IsChangePoint: Marked as a change because one of 6/22/2022 10:31:12 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -6.466117999484596 (T) = (0 -12591.240550799725) / Math.Sqrt((69596.82198162597 / (14)) + (79394.98687585884 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.06418279234267017 = (11831.840019778581 - 12591.240550799725) / 11831.840019778581 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json40KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 118.8626892992424 > 118.55053428507193.
IsChangePoint: Marked as a change because one of 6/17/2022 9:56:33 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -7.506023118864004 (T) = (0 -121572.12886191734) / Math.Sqrt((4192742.5510894423 / (14)) + (6482122.752315378 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.06649265512776761 = (113992.4670623754 - 121572.12886191734) / 113992.4670623754 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Format_MultipleArgs - Duration of single invocation 529.02 ns 850.06 ns 1.61 0.14 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.Format_MultipleArgs

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 850.0557408302284 > 547.6817332098481.
IsChangePoint: Marked as a change because one of 6/24/2022 10:40:04 AM, 6/27/2022 8:54:23 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -86.40554379355675 (T) = (0 -848.1242802325862) / Math.Sqrt((88.87602574889881 / (14)) + (69.21223243326581 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.6173858106947179 = (524.3796963127123 - 848.1242802325862) / 524.3796963127123 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Tests.Perf_Environment

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ExpandEnvironmentVariables - Duration of single invocation 382.06 ns 1.06 μs 2.76 0.11 False
SetEnvironmentVariable - Duration of single invocation 194.79 ns 661.73 ns 3.40 0.16 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Environment*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Environment.ExpandEnvironmentVariables

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.05539799053797 > 404.12774324999464.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -100.66329094480135 (T) = (0 -1058.1419394441955) / Math.Sqrt((68.03089949492843 / (14)) + (358.51775965172686 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -1.7472293516620847 = (385.16694603747425 - 1058.1419394441955) / 385.16694603747425 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Environment.SetEnvironmentVariable

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 661.725814258165 > 213.09369271099766.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -164.42300266168527 (T) = (0 -647.4959303229203) / Math.Sqrt((49.2398659816711 / (14)) + (35.11775717258779 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -2.2433183359130027 = (199.6399561379005 - 647.4959303229203) / 199.6399561379005 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Memory.ReadOnlySpan

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfString - Duration of single invocation 24.67 ns 36.76 ns 1.49 0.21 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.ReadOnlySpan*'

Payloads

Baseline
Compare

Histogram

System.Memory.ReadOnlySpan.IndexOfString(input: "StrIng", value: "string", comparisonType: OrdinalIgnoreCase)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.76482912607041 > 26.19735640294146.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -67.5375793028699 (T) = (0 -36.38052307472043) / Math.Sqrt((0.037995025208353364 / (14)) + (0.23599627187678154 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.4615325383200185 = (24.892037721266604 - 36.38052307472043) / 24.892037721266604 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in System.Collections.ContainsFalse<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ICollection - Duration of single invocation 186.65 μs 2.42 ms 12.95 0.01 False
Stack - Duration of single invocation 148.17 μs 2.98 ms 20.11 0.01 False
Queue - Duration of single invocation 186.46 μs 2.42 ms 12.98 0.02 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsFalse&lt;Int32&gt;*'
### Histogram

System.Collections.ContainsFalse<Int32>.ICollection(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.4180025503826528 > 196.43006513020836.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -373.14299335258266 (T) = (0 -2427206.700892857) / Math.Sqrt((1083311.6941798208 / (14)) + (323590191.29539 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -11.95486324505923 = (187358.7281454749 - 2427206.700892857) / 187358.7281454749 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsFalse&lt;Int32&gt;.Stack(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.9792487988095235 > 155.34581797022406.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -42.64814865820476 (T) = (0 -2532821.408927845) / Math.Sqrt((874624.8610895702 / (14)) + (28136948852.7768 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -16.092038104366907 = (148187.20818792962 - 2532821.408927845) / 148187.20818792962 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsFalse&lt;Int32&gt;.Queue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.4195034313186814 > 195.68580234160373.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -91.92109792877557 (T) = (0 -2470159.070670352) / Math.Sqrt((714329.7028732462 / (14)) + (5552447103.319803 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -12.216076602459486 = (186905.62600179392 - 2470159.070670352) / 186905.62600179392 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 543bcc5ee7d6a2b9471b016770227421c43a756e
Compare f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff Diff

Regressions in Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Load - Duration of single invocation 277.65 μs 294.38 μs 1.06 0.09 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks*'

Payloads

Baseline
Compare

Histogram

Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks.Load(FileName: "repeated.xml")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 294.3815035812672 > 286.28722408634746.
IsChangePoint: Marked as a change because one of 7/1/2022 12:02:05 PM, 7/10/2022 5:59:30 PM, 7/18/2022 3:18:54 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -9.519909182959154 (T) = (0 -300647.38905054185) / Math.Sqrt((42979330.934151694 / (14)) + (35851042.04585498 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.09181732773423353 = (275364.18539395486 - 300647.38905054185) / 275364.18539395486 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: naricc
Labels:

arch-wasm, untriaged, area-Codegen-AOT-mono, refs/heads/main, ubuntu 18.04, RunKind=micro, PGO, Regression, x64, AOT=true, WASM, CompliationMode=wasm
Milestone: -

@naricc naricc assigned radekdoulik and unassigned naricc Aug 24, 2022
@lewing
Copy link
Member

lewing commented Aug 24, 2022 •
edited
Loading

The span indexof regressions are pretty brutal

cc @marek-safar @vargaz

@lewing
Copy link
Member

lewing commented Aug 24, 2022

@adamsitnik we think this might be related to the Vector128 changes

@lewing lewing removed the untriaged New issue has not been triaged by the area owner label Aug 24, 2022
@lewing lewing modified the milestones: 7.0.0, 8.0.0 Sep 14, 2022
@naricc naricc mentioned this issue Sep 15, 2022
Closed
@lewing
Copy link
Member

lewing commented Sep 15, 2022 •
edited
Loading

Also regressed the mono interpreter pretty badly dotnet/perf-autofiling-issues#7976

cc @BrzVlad @vargaz

@lewing
Copy link
Member

lewing commented Sep 15, 2022 •
edited
Loading

we can see the impact of #73768 very clearly in other aot json tests but the span here is large in perf pipeline and the other vectorization might have caused issues as well.

@vargaz
Copy link
Contributor

vargaz commented Sep 15, 2022 •
edited
Loading

The usage of generic structs like:

private static int IndexOfAnyValueType<TValue, TNegator>(ref TValue searchSpace, TValue value0, TValue value1, TValue value2, TValue value3, int length)
            where TValue : struct, INumber<TValue>
            where TNegator : struct, INegator<TValue>

probably causes wasm to fall back to the interpreter for these methods, since the AOT compiler cannot determine the instantiations that will be used at runtime.

Imho such code should be avoided in low level BCL code, its better to have some code duplication.

@SamMonoRT SamMonoRT mentioned this issue Sep 15, 2022
Merged
@tannergooding
Copy link
Member

tannergooding commented Sep 15, 2022 •
edited
Loading

Imho such code should be avoided in low level BCL code, its better to have some code duplication.

This isn't really practical, IMO, for us or our customers who are going to do the same. We have many features that are built on and around generics and generic specialization.

Much like with Vector<T>, Vector64<T>, Vector128<T>, and Vector256<T> this really seems like something that WASM and AOT will have to handle as they are such core concepts to .NET as a whole (especially in perf oriented code).

Many of these algorithms are very long and very complex. It's not just a small amount of duplication, but large amounts of duplication in some of our most critical code (where having one place to review/update is often core/critical due to its complexity).

@tannergooding
Copy link
Member

tannergooding commented Sep 15, 2022 •
edited
Loading

Notably, in the case of IndexOfAnyValueType, these are all "internal" generic methods that are called from other methods where the type is concrete and well-known.

WASM/AOT could recognize the pattern of if (typeof(T) == typeof(...)) for value types and/or the cases where we are calling with concrete type information and either specialize or inline the methods to ensure it doesn't need to interpret (e.g. https://source.dot.net/#System.Private.CoreLib/src/libraries/System.Private.CoreLib/src/System/MemoryExtensions.cs,1162 has a size check and then always calls the API as IndexOfAnyValueType<byte>).

@lewing
Copy link
Member

lewing commented Sep 16, 2022

in the mean time string handling is now slower than net6 in many cases and it happened post the rc1 branch

@lewing
Copy link
Member

lewing commented Sep 16, 2022 •
edited
Loading

cc @danmoseley

#74086

@tannergooding
Copy link
Member

Notably in WASM. Most other targets, including Arm64, have gotten significantly faster (often 2x or more).

The “best” fix for .NET 7 given that and the short timeframe is likely to ifdef for WASM or similar. I don’t think reverting it is the right choice given the gains we see on almost all our other targets.

For .NET 8 I think this really needs a more in depth look from the WASM side on how generics can be better supported. That could be one of the suggestions above or something as “simple” as some attribute that tells WASM/AOT what to specialize for in an opt-in manner. I don’t believe duplicating code or maintaining the status quo is viable, especially given the prevalence of genetics in .NET and all the newer features and existing perf patterns that encourage users to write generic and generic specialized algorithms

@lewing lewing mentioned this issue Sep 16, 2022
Closed
@danmoseley
Copy link
Member

Historically when doing this kind of vectorization work we have assumed it's reasonable to measure only on CoreCLR x64 and Arm64 and I assume that's what @adamsitnik did. Maybe it's fine to rely on the perf lab to catch anything Mono specific - it's just that timeframes are tighter at the moment.

If #if is feasible that seems reasonable as me in the short term?

@jkotas
Copy link
Member

jkotas commented Sep 16, 2022

the AOT compiler cannot determine the instantiations that will be used at runtime.

@vargaz Why is the AOT compiler not able to determine the instantiations that will be used at runtime in this case?

The instantiation chain for these generic helper methods always uses a small set of concrete types. There is no ambiguity about what this will be instantiated with.

@vargaz
Copy link
Contributor

vargaz commented Sep 16, 2022

So what happens is that the AOT compiler creates a shared instantiation for bytes and enums whose base type is bytes, and that ends up being called, and then it cannot inline the call to DontNegate.NegateIfNeeded () into that. This appears fixable, but not in the net7 timeframe.

@adamsitnik
Copy link
Member

This appears fixable, but not in the net7 timeframe.

@jkotas @vargaz should I port the fix to just release/7.0-rc2 branch or both main and release/7.0-rc2?

@vargaz
Copy link
Contributor

vargaz commented Sep 16, 2022

I think both if possible.

@stephentoub
Copy link
Member

We should start with a fix in main and then backport it to rc2.

@SamMonoRT SamMonoRT modified the milestones: 8.0.0, 7.0.0 Sep 19, 2022
@jeffhandley jeffhandley mentioned this issue Sep 20, 2022
Merged
2 tasks
@lewing
Copy link
Member

lewing commented Sep 26, 2022

resolved for net7 in #75996

@lewing lewing closed this as completed Sep 26, 2022
@EgorBo EgorBo mentioned this issue Sep 28, 2022
Closed
18 tasks
@ghost ghost locked as resolved and limited conversation to collaborators Oct 27, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@radekdoulik radekdoulik

@adamsitnik adamsitnik

Labels
arch-wasm WebAssembly architecture area-Codegen-AOT-mono blocking-release
Projects
None yet
Milestone
7.0.0
Development

No branches or pull requests