[Perf] Linux/arm64: 4 Regressions on 3/28/2023 2:42:21 AM #14633
Description
Run Information
| Name | Value |
|---|---|
| Architecture | arm64 |
| OS | ubuntu 20.04 |
| Queue | AmpereUbuntu |
| Baseline | 8d5f520838bbe2fe0ad02df53c85809fbbb1fcbb |
| Compare | 3231e4fbfbef268b1647bf6f695e6b42cfb2117d |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Count - Duration of single invocation | 4.33 ms | 4.82 ms | 1.11 | 0.35 | False | |||||
| Count - Duration of single invocation | 4.26 ms | 4.87 ms | 1.14 | 0.39 | False | |||||
| Count - Duration of single invocation | 12.80 ms | 16.83 ms | 1.31 | 0.04 | True | |||||
| Count - Duration of single invocation | 16.84 ms | 21.23 ms | 1.26 | 0.01 | True |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\w+\s+Holmes\s+\w+", Options: Compiled)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.819193994791667 > 4.552229026293103.
IsChangePoint: Marked as a change because one of 3/27/2023 3:21:55 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsRegressionStdDev: Marked as regression because -12.756492135980336 (T) = (0 -4795824.034725765) / Math.Sqrt((27922608730.92018 / (31)) + (154549921.9120878 / (7))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (7) - 2, .025) and -0.0879052347587299 = (4408310.468134992 - 4795824.034725765) / 4408310.468134992 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\w+\s+Holmes", Options: Compiled)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.872532517857143 > 4.482539235905904.
IsChangePoint: Marked as a change because one of 12/25/2022 11:45:49 PM, 12/27/2022 1:08:06 PM, 3/25/2023 2:21:39 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsRegressionStdDev: Marked as regression because -3.2418812551457306 (T) = (0 -4618381.482173909) / Math.Sqrt((23252746055.079617 / (23)) + (101193558559.1289 / (15))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (15) - 2, .025) and -0.06589863743060032 = (4332852.412033042 - 4618381.482173909) / 4332852.412033042 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\b\w+n\b", Options: Compiled)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.82856401333333 > 13.444227299596152.
IsChangePoint: Marked as a change because one of 3/28/2023 2:42:21 AM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsRegressionStdDev: Marked as regression because -24.027129135169304 (T) = (0 -16720176.586133787) / Math.Sqrt((761750069493.8451 / (33)) + (4225525173.170076 / (6))) is less than -2.026192463026769 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (6) - 2, .025) and -0.28474253276079076 = (13014418.188680733 - 16720176.586133787) / 13014418.188680733 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{L}", Options: Compiled)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.2263869125 > 17.688542270000003.
IsChangePoint: Marked as a change because one of 3/28/2023 2:42:21 AM, 3/30/2023 4:24:24 AM falls between 3/21/2023 2:53:21 PM and 3/30/2023 4:24:24 AM.
IsRegressionStdDev: Marked as regression because -78.14928473949419 (T) = (0 -21087444.902083334) / Math.Sqrt((19916808307.44899 / (33)) + (13728388400.888712 / (6))) is less than -2.026192463026769 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (6) - 2, .025) and -0.24888085607666177 = (16885073.383484464 - 21087444.902083334) / 16885073.383484464 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository