Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Changes at 5/28/2022 1:41:21 AM #70162

Closed
performanceautofiler bot opened this issue Jun 2, 2022 · 5 comments
Closed

[Perf] Changes at 5/28/2022 1:41:21 AM #70162

performanceautofiler bot opened this issue Jun 2, 2022 · 5 comments
Assignees
@tannergooding
Labels
arch-arm64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI runtime-coreclr specific to the CoreCLR runtime
Milestone
7.0.0

Comments

@performanceautofiler
Copy link

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 7f7c6a229b691bd060ee1bf4aeb2cea4c5e52168
Compare 748dfb78ef87b38599b89e85d3201cdb9f9c0b33
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 2.20 ms 2.98 ms 1.36 0.05 True
Count - Duration of single invocation 153.31 μs 191.44 μs 1.25 0.01 True
Count - Duration of single invocation 1.24 ms 1.37 ms 1.10 0.10 True
Count - Duration of single invocation 24.70 ms 34.02 ms 1.38 0.10 True
Count - Duration of single invocation 144.68 μs 165.27 μs 1.14 0.01 False
Count - Duration of single invocation 240.75 μs 269.14 μs 1.12 0.01 True
Count - Duration of single invocation 12.03 ms 21.39 ms 1.78 0.05 True
Count - Duration of single invocation 47.29 ms 59.27 ms 1.25 0.44 True
Count - Duration of single invocation 219.57 μs 232.72 μs 1.06 0.01 True
Count - Duration of single invocation 89.40 μs 123.35 μs 1.38 0.01 True
Count - Duration of single invocation 2.33 ms 2.46 ms 1.06 0.02 True
Count - Duration of single invocation 177.12 μs 211.80 μs 1.20 0.01 True
Count - Duration of single invocation 3.18 ms 3.36 ms 1.06 0.02 True
Count - Duration of single invocation 3.08 ms 3.48 ms 1.13 0.13 True
Count - Duration of single invocation 94.23 μs 128.22 μs 1.36 0.01 True
Count - Duration of single invocation 374.51 μs 422.05 μs 1.13 0.01 True
Count - Duration of single invocation 49.23 ms 60.94 ms 1.24 0.48 True

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the\s+\w+", Options: NonBacktracking)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.981663075502008 > 2.30941796713219.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -28.080765181910134 (T) = (0 -2748213.418441032) / Math.Sqrt((563613961.9919467 / (25)) + (7259741238.014956 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.2509798087203997 = (2196848.7415093617 - 2748213.418441032) / 2196848.7415093617 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 191.43783472206997 > 160.95435806566604.
IsChangePoint: Marked as a change because one of 5/31/2022 5:02:26 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -77.0805759253954 (T) = (0 -190973.89435431754) / Math.Sqrt((6394852.873722199 / (39)) + (317440.2113534399 / (6))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (6) - 2, .025) and -0.23147466580807602 = (155077.40407229832 - 190973.89435431754) / 155077.40407229832 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3717235683241253 > 1.2718615059933995.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -18.20562778818909 (T) = (0 -1386204.0048313157) / Math.Sqrt((627918840.8777322 / (25)) + (1547500831.444926 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.15335016285121025 = (1201893.4487376017 - 1386204.0048313157) / 1201893.4487376017 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\w+", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.020709333333336 > 26.112056162500004.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -23.71893120000806 (T) = (0 -33687196.72236111) / Math.Sqrt((108679716720.46556 / (25)) + (2882756164569.5947 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.3723324199439214 = (24547402.825138893 - 33687196.72236111) / 24547402.825138893 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 165.2708110011001 > 151.98199722719656.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 5/31/2022 5:02:26 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -24.825963965192063 (T) = (0 -166486.91418455934) / Math.Sqrt((34070055.2268123 / (39)) + (1392632.58256889 / (6))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (6) - 2, .025) and -0.1859572100506226 = (140381.88964461276 - 166486.91418455934) / 140381.88964461276 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "The", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 269.14373965358675 > 252.81119025480768.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -17.8257511259632 (T) = (0 -263047.12470040156) / Math.Sqrt((10901377.745342152 / (25)) + (20715833.607267246 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.08957857224984643 = (241420.97816520146 - 263047.12470040156) / 241420.97816520146 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?s).*", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.390846266666667 > 12.653760510955129.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -29.408494072301586 (T) = (0 -21969351.4441687) / Math.Sqrt((4309104161.758038 / (25)) + (2284619509591.901 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.8273756474998881 = (12022351.000587057 - 21969351.4441687) / 12022351.000587057 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{L}", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 59.271682283333334 > 52.54472064947369.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -19.252701802540358 (T) = (0 -60574629.67997309) / Math.Sqrt((5246988632455.257 / (25)) + (3218294459578.9824 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.23998483329436424 = (48851105.314764015 - 60574629.67997309) / 48851105.314764015 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Holmes", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 232.71666604477616 > 231.6408227692195.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -35.65556806119108 (T) = (0 -233717.0214530257) / Math.Sqrt((1794347.206209707 / (25)) + (1702926.0782103387 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.06432041301297259 = (219592.72658446795 - 233717.0214530257) / 219592.72658446795 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 123.34520114829395 > 93.96958350142047.
IsChangePoint: Marked as a change because one of 5/31/2022 5:02:26 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -215.78352088181776 (T) = (0 -123009.64032435244) / Math.Sqrt((94771.9133460141 / (39)) + (131928.7193142191 / (6))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (6) - 2, .025) and -0.3776321281885864 = (89290.62977508713 - 123009.64032435244) / 89290.62977508713 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)the", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.464413328712871 > 2.403919837526334.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -16.011236332799633 (T) = (0 -2488036.238172627) / Math.Sqrt((1714946164.887705 / (25)) + (1986602740.6466775 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.09098012790848044 = (2280551.381758387 - 2488036.238172627) / 2280551.381758387 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 211.80437533783785 > 185.6900772735253.
IsChangePoint: Marked as a change because one of 5/31/2022 5:02:26 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -146.412602916086 (T) = (0 -211632.1476028529) / Math.Sqrt((310405.9975140347 / (39)) + (288637.8608624683 / (6))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (6) - 2, .025) and -0.1959026234001499 = (176964.36437370421 - 211632.1476028529) / 176964.36437370421 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes|Watson|Irene|Adler|John|Baker", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.3579610954954955 > 3.3373258117721516.
IsChangePoint: Marked as a change because one of 5/23/2022 1:47:22 PM, 5/30/2022 4:33:23 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -4.01558145739983 (T) = (0 -3419948.6696357788) / Math.Sqrt((1361121048.566335 / (35)) + (40171019006.81633 / (10))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (10) - 2, .025) and -0.08082238832378143 = (3164209.685681739 - 3419948.6696357788) / 3164209.685681739 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{Lu}", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.477458443452381 > 3.1315628058333336.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -19.748340139107853 (T) = (0 -3432711.8964723423) / Math.Sqrt((5397452138.060062 / (25)) + (8744618333.829802 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.1723680192829514 = (2928015.640150157 - 3432711.8964723423) / 2928015.640150157 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 128.21670056352457 > 99.13207817865211.
IsChangePoint: Marked as a change because one of 5/31/2022 5:02:26 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -181.94302304862808 (T) = (0 -128889.10613489995) / Math.Sqrt((94724.04877640512 / (39)) + (204466.33468793487 / (6))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (6) - 2, .025) and -0.3693277797697027 = (94125.8243928834 - 128889.10613489995) / 94125.8243928834 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sher[a-z]+|Hol[a-z]+", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 422.0465741242937 > 397.00675627836614.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -18.176481339885193 (T) = (0 -423290.6777829517) / Math.Sqrt((14052835.688902048 / (25)) + (123721501.26211147 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.1255541612870132 = (376073.13121115434 - 423290.6777829517) / 376073.13121115434 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{Ll}", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 60.93665580263158 > 48.91599193031251.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -19.59316576124937 (T) = (0 -59190504.20009433) / Math.Sqrt((3398672357602.4917 / (25)) + (3952651967907.543 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.2363756179036488 = (47874208.56814977 - 59190504.20009433) / 47874208.56814977 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline 7f7c6a229b691bd060ee1bf4aeb2cea4c5e52168
Compare 748dfb78ef87b38599b89e85d3201cdb9f9c0b33
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsMatch - Duration of single invocation 309.47 ns 405.39 ns 1.31 0.05 True
IsMatch - Duration of single invocation 125.32 ns 150.51 ns 1.20 0.07 True
IsMatch - Duration of single invocation 202.59 ns 225.33 ns 1.11 0.12 False
IsMatch - Duration of single invocation 288.86 ns 305.95 ns 1.06 0.07 True
IsMatch - Duration of single invocation 203.85 ns 233.80 ns 1.15 0.07 False
IsMatch - Duration of single invocation 161.87 ns 240.99 ns 1.49 0.30 True
IsMatch - Duration of single invocation 183.98 ns 201.75 ns 1.10 0.07 True
IsMatch - Duration of single invocation 229.50 ns 258.29 ns 1.13 0.10 False
IsMatch - Duration of single invocation 194.10 ns 216.62 ns 1.12 0.07 False
IsMatch - Duration of single invocation 284.58 ns 324.36 ns 1.14 0.11 True

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 5, Options: NonBacktracking)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 405.3857795166146 > 320.66215880895163.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -40.418647382839566 (T) = (0 -411.15346009345296) / Math.Sqrt((32.35787412099058 / (25)) + (117.72592224435641 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.3576367213868147 = (302.84497584409996 - 411.15346009345296) / 302.84497584409996 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 0, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 150.50710559372936 > 132.35765893001678.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -16.603267025237894 (T) = (0 -156.93849754400017) / Math.Sqrt((23.36308685349803 / (25)) + (45.54719564086743 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.23396226473638665 = (127.18257440192241 - 156.93849754400017) / 127.18257440192241 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 9, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 225.3326525326946 > 216.42381707591662.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -5.17397456347185 (T) = (0 -229.7646021412044) / Math.Sqrt((330.28200735519516 / (25)) + (28.338981077832774 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.09424340910274114 = (209.97576976918418 - 229.7646021412044) / 209.97576976918418 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 2, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 305.9530747031089 > 303.2243229249164.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -17.965470195742096 (T) = (0 -314.4026584100449) / Math.Sqrt((27.614845224254083 / (25)) + (27.37611344161136 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.0987403306997623 = (286.14828237879385 - 314.4026584100449) / 286.14828237879385 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 6, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 233.79921100406875 > 216.0981848457079.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -9.520391677916491 (T) = (0 -235.10568109278287) / Math.Sqrt((17.71853965226845 / (25)) + (172.63511589388332 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.14123850434528068 = (206.0092436397956 - 235.10568109278287) / 206.0092436397956 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 11, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 240.9901997942921 > 168.67270178875302.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -37.06145896087716 (T) = (0 -235.74694942321312) / Math.Sqrt((20.544573306682103 / (25)) + (63.292732744651765 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.45747758142672895 = (161.749966124652 - 235.74694942321312) / 161.749966124652 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 7, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 201.74684363474523 > 192.63635665023534.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -19.338747961834603 (T) = (0 -208.1492449451767) / Math.Sqrt((16.669059602977708 / (25)) + (22.631321600765006 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.14232369415752283 = (182.21564168699942 - 208.1492449451767) / 182.21564168699942 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 10, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 258.2913743473124 > 240.22440418405336.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -7.347499804753521 (T) = (0 -255.74952059699817) / Math.Sqrt((225.26819447156268 / (25)) + (28.97166097012453 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.10243003638235425 = (231.98707596560524 - 255.74952059699817) / 231.98707596560524 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 8, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 216.62043721607395 > 210.04413335180266.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -7.4035224375679105 (T) = (0 -217.32059219734487) / Math.Sqrt((181.32903565283027 / (25)) + (20.14139467969238 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.10853866158707445 = (196.04241126440368 - 217.32059219734487) / 196.04241126440368 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 12, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 324.3579286835487 > 288.0388752258521.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -21.307866588903774 (T) = (0 -317.91077286602257) / Math.Sqrt((60.57126691214733 / (25)) + (31.936782298424202 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.1552398254605527 = (275.1902815844173 - 317.91077286602257) / 275.1902815844173 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added arm64 untriaged New issue has not been triaged by the area owner labels Jun 2, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@DrewScoggins DrewScoggins transferred this issue from dotnet/perf-autofiling-issues Jun 2, 2022
@DrewScoggins
Copy link
Member

Changes that occurred on May 31st are most likey related to #68874

We are also seeing changes on May 27th in this issue, and this is the range of commits 1f303a1...77520a5

We are not sure which of these commits is the offender.

@stephentoub
Copy link
Member

Dup of #70160?

@jeffschwMSFT jeffschwMSFT added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Jun 3, 2022
@ghost
Copy link

ghost commented Jun 3, 2022

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 7f7c6a229b691bd060ee1bf4aeb2cea4c5e52168
Compare 748dfb78ef87b38599b89e85d3201cdb9f9c0b33
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 2.20 ms 2.98 ms 1.36 0.05 True
Count - Duration of single invocation 153.31 μs 191.44 μs 1.25 0.01 True
Count - Duration of single invocation 1.24 ms 1.37 ms 1.10 0.10 True
Count - Duration of single invocation 24.70 ms 34.02 ms 1.38 0.10 True
Count - Duration of single invocation 144.68 μs 165.27 μs 1.14 0.01 False
Count - Duration of single invocation 240.75 μs 269.14 μs 1.12 0.01 True
Count - Duration of single invocation 12.03 ms 21.39 ms 1.78 0.05 True
Count - Duration of single invocation 47.29 ms 59.27 ms 1.25 0.44 True
Count - Duration of single invocation 219.57 μs 232.72 μs 1.06 0.01 True
Count - Duration of single invocation 89.40 μs 123.35 μs 1.38 0.01 True
Count - Duration of single invocation 2.33 ms 2.46 ms 1.06 0.02 True
Count - Duration of single invocation 177.12 μs 211.80 μs 1.20 0.01 True
Count - Duration of single invocation 3.18 ms 3.36 ms 1.06 0.02 True
Count - Duration of single invocation 3.08 ms 3.48 ms 1.13 0.13 True
Count - Duration of single invocation 94.23 μs 128.22 μs 1.36 0.01 True
Count - Duration of single invocation 374.51 μs 422.05 μs 1.13 0.01 True
Count - Duration of single invocation 49.23 ms 60.94 ms 1.24 0.48 True

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the\s+\w+", Options: NonBacktracking)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.981663075502008 > 2.30941796713219.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -28.080765181910134 (T) = (0 -2748213.418441032) / Math.Sqrt((563613961.9919467 / (25)) + (7259741238.014956 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.2509798087203997 = (2196848.7415093617 - 2748213.418441032) / 2196848.7415093617 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 191.43783472206997 > 160.95435806566604.
IsChangePoint: Marked as a change because one of 5/31/2022 5:02:26 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -77.0805759253954 (T) = (0 -190973.89435431754) / Math.Sqrt((6394852.873722199 / (39)) + (317440.2113534399 / (6))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (6) - 2, .025) and -0.23147466580807602 = (155077.40407229832 - 190973.89435431754) / 155077.40407229832 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3717235683241253 > 1.2718615059933995.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -18.20562778818909 (T) = (0 -1386204.0048313157) / Math.Sqrt((627918840.8777322 / (25)) + (1547500831.444926 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.15335016285121025 = (1201893.4487376017 - 1386204.0048313157) / 1201893.4487376017 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\w+", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.020709333333336 > 26.112056162500004.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -23.71893120000806 (T) = (0 -33687196.72236111) / Math.Sqrt((108679716720.46556 / (25)) + (2882756164569.5947 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.3723324199439214 = (24547402.825138893 - 33687196.72236111) / 24547402.825138893 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 165.2708110011001 > 151.98199722719656.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 5/31/2022 5:02:26 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -24.825963965192063 (T) = (0 -166486.91418455934) / Math.Sqrt((34070055.2268123 / (39)) + (1392632.58256889 / (6))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (6) - 2, .025) and -0.1859572100506226 = (140381.88964461276 - 166486.91418455934) / 140381.88964461276 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "The", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 269.14373965358675 > 252.81119025480768.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -17.8257511259632 (T) = (0 -263047.12470040156) / Math.Sqrt((10901377.745342152 / (25)) + (20715833.607267246 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.08957857224984643 = (241420.97816520146 - 263047.12470040156) / 241420.97816520146 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?s).*", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.390846266666667 > 12.653760510955129.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -29.408494072301586 (T) = (0 -21969351.4441687) / Math.Sqrt((4309104161.758038 / (25)) + (2284619509591.901 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.8273756474998881 = (12022351.000587057 - 21969351.4441687) / 12022351.000587057 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{L}", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 59.271682283333334 > 52.54472064947369.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -19.252701802540358 (T) = (0 -60574629.67997309) / Math.Sqrt((5246988632455.257 / (25)) + (3218294459578.9824 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.23998483329436424 = (48851105.314764015 - 60574629.67997309) / 48851105.314764015 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Holmes", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 232.71666604477616 > 231.6408227692195.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -35.65556806119108 (T) = (0 -233717.0214530257) / Math.Sqrt((1794347.206209707 / (25)) + (1702926.0782103387 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.06432041301297259 = (219592.72658446795 - 233717.0214530257) / 219592.72658446795 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 123.34520114829395 > 93.96958350142047.
IsChangePoint: Marked as a change because one of 5/31/2022 5:02:26 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -215.78352088181776 (T) = (0 -123009.64032435244) / Math.Sqrt((94771.9133460141 / (39)) + (131928.7193142191 / (6))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (6) - 2, .025) and -0.3776321281885864 = (89290.62977508713 - 123009.64032435244) / 89290.62977508713 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)the", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.464413328712871 > 2.403919837526334.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -16.011236332799633 (T) = (0 -2488036.238172627) / Math.Sqrt((1714946164.887705 / (25)) + (1986602740.6466775 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.09098012790848044 = (2280551.381758387 - 2488036.238172627) / 2280551.381758387 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 211.80437533783785 > 185.6900772735253.
IsChangePoint: Marked as a change because one of 5/31/2022 5:02:26 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -146.412602916086 (T) = (0 -211632.1476028529) / Math.Sqrt((310405.9975140347 / (39)) + (288637.8608624683 / (6))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (6) - 2, .025) and -0.1959026234001499 = (176964.36437370421 - 211632.1476028529) / 176964.36437370421 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes|Watson|Irene|Adler|John|Baker", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.3579610954954955 > 3.3373258117721516.
IsChangePoint: Marked as a change because one of 5/23/2022 1:47:22 PM, 5/30/2022 4:33:23 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -4.01558145739983 (T) = (0 -3419948.6696357788) / Math.Sqrt((1361121048.566335 / (35)) + (40171019006.81633 / (10))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (10) - 2, .025) and -0.08082238832378143 = (3164209.685681739 - 3419948.6696357788) / 3164209.685681739 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{Lu}", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.477458443452381 > 3.1315628058333336.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -19.748340139107853 (T) = (0 -3432711.8964723423) / Math.Sqrt((5397452138.060062 / (25)) + (8744618333.829802 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.1723680192829514 = (2928015.640150157 - 3432711.8964723423) / 2928015.640150157 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 128.21670056352457 > 99.13207817865211.
IsChangePoint: Marked as a change because one of 5/31/2022 5:02:26 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -181.94302304862808 (T) = (0 -128889.10613489995) / Math.Sqrt((94724.04877640512 / (39)) + (204466.33468793487 / (6))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (6) - 2, .025) and -0.3693277797697027 = (94125.8243928834 - 128889.10613489995) / 94125.8243928834 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sher[a-z]+|Hol[a-z]+", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 422.0465741242937 > 397.00675627836614.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -18.176481339885193 (T) = (0 -423290.6777829517) / Math.Sqrt((14052835.688902048 / (25)) + (123721501.26211147 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.1255541612870132 = (376073.13121115434 - 423290.6777829517) / 376073.13121115434 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{Ll}", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 60.93665580263158 > 48.91599193031251.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -19.59316576124937 (T) = (0 -59190504.20009433) / Math.Sqrt((3398672357602.4917 / (25)) + (3952651967907.543 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.2363756179036488 = (47874208.56814977 - 59190504.20009433) / 47874208.56814977 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline 7f7c6a229b691bd060ee1bf4aeb2cea4c5e52168
Compare 748dfb78ef87b38599b89e85d3201cdb9f9c0b33
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsMatch - Duration of single invocation 309.47 ns 405.39 ns 1.31 0.05 True
IsMatch - Duration of single invocation 125.32 ns 150.51 ns 1.20 0.07 True
IsMatch - Duration of single invocation 202.59 ns 225.33 ns 1.11 0.12 False
IsMatch - Duration of single invocation 288.86 ns 305.95 ns 1.06 0.07 True
IsMatch - Duration of single invocation 203.85 ns 233.80 ns 1.15 0.07 False
IsMatch - Duration of single invocation 161.87 ns 240.99 ns 1.49 0.30 True
IsMatch - Duration of single invocation 183.98 ns 201.75 ns 1.10 0.07 True
IsMatch - Duration of single invocation 229.50 ns 258.29 ns 1.13 0.10 False
IsMatch - Duration of single invocation 194.10 ns 216.62 ns 1.12 0.07 False
IsMatch - Duration of single invocation 284.58 ns 324.36 ns 1.14 0.11 True

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 5, Options: NonBacktracking)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 405.3857795166146 > 320.66215880895163.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -40.418647382839566 (T) = (0 -411.15346009345296) / Math.Sqrt((32.35787412099058 / (25)) + (117.72592224435641 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.3576367213868147 = (302.84497584409996 - 411.15346009345296) / 302.84497584409996 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 0, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 150.50710559372936 > 132.35765893001678.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -16.603267025237894 (T) = (0 -156.93849754400017) / Math.Sqrt((23.36308685349803 / (25)) + (45.54719564086743 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.23396226473638665 = (127.18257440192241 - 156.93849754400017) / 127.18257440192241 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 9, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 225.3326525326946 > 216.42381707591662.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -5.17397456347185 (T) = (0 -229.7646021412044) / Math.Sqrt((330.28200735519516 / (25)) + (28.338981077832774 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.09424340910274114 = (209.97576976918418 - 229.7646021412044) / 209.97576976918418 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 2, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 305.9530747031089 > 303.2243229249164.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -17.965470195742096 (T) = (0 -314.4026584100449) / Math.Sqrt((27.614845224254083 / (25)) + (27.37611344161136 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.0987403306997623 = (286.14828237879385 - 314.4026584100449) / 286.14828237879385 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 6, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 233.79921100406875 > 216.0981848457079.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -9.520391677916491 (T) = (0 -235.10568109278287) / Math.Sqrt((17.71853965226845 / (25)) + (172.63511589388332 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.14123850434528068 = (206.0092436397956 - 235.10568109278287) / 206.0092436397956 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 11, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 240.9901997942921 > 168.67270178875302.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -37.06145896087716 (T) = (0 -235.74694942321312) / Math.Sqrt((20.544573306682103 / (25)) + (63.292732744651765 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.45747758142672895 = (161.749966124652 - 235.74694942321312) / 161.749966124652 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 7, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 201.74684363474523 > 192.63635665023534.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -19.338747961834603 (T) = (0 -208.1492449451767) / Math.Sqrt((16.669059602977708 / (25)) + (22.631321600765006 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.14232369415752283 = (182.21564168699942 - 208.1492449451767) / 182.21564168699942 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 10, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 258.2913743473124 > 240.22440418405336.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -7.347499804753521 (T) = (0 -255.74952059699817) / Math.Sqrt((225.26819447156268 / (25)) + (28.97166097012453 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.10243003638235425 = (231.98707596560524 - 255.74952059699817) / 231.98707596560524 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 8, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 216.62043721607395 > 210.04413335180266.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -7.4035224375679105 (T) = (0 -217.32059219734487) / Math.Sqrt((181.32903565283027 / (25)) + (20.14139467969238 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.10853866158707445 = (196.04241126440368 - 217.32059219734487) / 196.04241126440368 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 12, Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 324.3579286835487 > 288.0388752258521.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -21.307866588903774 (T) = (0 -317.91077286602257) / Math.Sqrt((60.57126691214733 / (25)) + (31.936782298424202 / (20))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (20) - 2, .025) and -0.1552398254605527 = (275.1902815844173 - 317.91077286602257) / 275.1902815844173 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: tannergooding
Labels:

area-CodeGen-coreclr, untriaged, refs/heads/main, RunKind=micro, Regression, CoreClr, arm64, ubuntu 20.04
Milestone: -

@JulieLeeMSFT JulieLeeMSFT removed the untriaged New issue has not been triaged by the area owner label Jun 3, 2022
@JulieLeeMSFT JulieLeeMSFT added this to the 7.0.0 milestone Jun 3, 2022
@AndyAyersMS AndyAyersMS mentioned this issue Jun 7, 2022
Closed
@tannergooding
Copy link
Member

Most of these were resolved in 2d5c8dc

A few were mostly fixed but still have a small regression related to #70182

@ghost ghost locked as resolved and limited conversation to collaborators Jul 17, 2022
@jeffhandley jeffhandley added arch-arm64 runtime-coreclr specific to the CoreCLR runtime and removed arm64 labels Dec 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@tannergooding tannergooding

Labels
arch-arm64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI runtime-coreclr specific to the CoreCLR runtime
Projects
None yet
Milestone
7.0.0
Development

No branches or pull requests
6 participants
@jeffhandley @stephentoub @DrewScoggins @jeffschwMSFT @tannergooding @JulieLeeMSFT