Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

[performanceautofiler]

Run Information

Architecture	arm64
OS	ubuntu 20.04
Baseline	feb9968ad754c7116178fdc3216dc12977bebbf0
Compare	72ad6649b505a98cae5ff296818df2d7a5da4807
Diff	Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Count - Duration of single invocation	37.70 ms	42.81 ms	1.14	0.02	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'

Related Issues

Regressions

Improvements

• [Perf] Windows/x64: 2 Improvements on 11/10/2022 12:18:51 PM
• [Perf] Linux/x64: 3 Improvements on 11/10/2022 12:18:51 PM

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "\p{Sm}", Options: NonBacktracking)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 42.807339102040814 > 39.579856905.
IsChangePoint: Marked as a change because one of 10/27/2022 12:34:03 AM, 11/9/2022 4:06:23 PM, 11/16/2022 10:12:48 PM falls between 11/8/2022 2:29:58 AM and 11/16/2022 10:12:48 PM.
IsRegressionStdDev: Marked as regression because -2438.7226467982086 (T) = (0 -42806637.74850602) / Math.Sqrt((62350313.25567417 / (16)) + (10043082.165012123 / (20))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (20) - 2, .025) and -0.13570513007489693 = (37691683.003741495 - 42806637.74850602) / 37691683.003741495 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[LoopedBard3]

@stephentoub we are guessing this regression was caused by one of these:
#78003
#78103
#78130

Also related regression: dotnet/perf-autofiling-issues#9847

[stephentoub]

Is there actually a regression here? Seems like it briefly dipped for a few days but then went back to exactly where it was before?

[ghost]

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run Information

Architecture	arm64
OS	ubuntu 20.04
Baseline	feb9968ad754c7116178fdc3216dc12977bebbf0
Compare	72ad6649b505a98cae5ff296818df2d7a5da4807
Diff	Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Count - Duration of single invocation	37.70 ms	42.81 ms	1.14	0.02	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'

Related Issues

Regressions

Improvements

• [Perf] Windows/x64: 2 Improvements on 11/10/2022 12:18:51 PM
• [Perf] Linux/x64: 3 Improvements on 11/10/2022 12:18:51 PM

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "\p{Sm}", Options: NonBacktracking)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 42.807339102040814 > 39.579856905.
IsChangePoint: Marked as a change because one of 10/27/2022 12:34:03 AM, 11/9/2022 4:06:23 PM, 11/16/2022 10:12:48 PM falls between 11/8/2022 2:29:58 AM and 11/16/2022 10:12:48 PM.
IsRegressionStdDev: Marked as regression because -2438.7226467982086 (T) = (0 -42806637.74850602) / Math.Sqrt((62350313.25567417 / (16)) + (10043082.165012123 / (20))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (20) - 2, .025) and -0.13570513007489693 = (37691683.003741495 - 42806637.74850602) / 37691683.003741495 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	performanceautofiler[bot]
Assignees:	-
Labels:	area-System.Text.RegularExpressions, untriaged
Milestone:	-

[LoopedBard3]

Looking at the drop, there were some changes to the Regex section (f1a093f). I am not sure if the changes should have resulted in an improvement. It looks like most of the regressions in the related regression issue did the same down then up to the same position, except for the System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock regression which ended up higher than the previous line. Generally, I am unfamiliar with this area so will take what someone with more knowledge thinks.

[stephentoub]

Thanks, @LoopedBard3. I took a deeper look, and I think the remaining tiny regressions from #78003 are acceptable fallout from fixing the other issues.