Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Perf] Linux/x64: 2 Regressions on 12/5/2022 7:07:15 PM #79604

Closed
performanceautofiler bot opened this issue Dec 13, 2022 · 4 comments
Closed

[Perf] Linux/x64: 2 Regressions on 12/5/2022 7:07:15 PM #79604

performanceautofiler bot opened this issue Dec 13, 2022 · 4 comments
Assignees
Labels
Milestone

Comments

@performanceautofiler
Copy link

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 315f4ab59adf7eaec815885825c5b0d65fc3ab99
Compare 21a3647f09ca92cbf890129d4e917cbd01a1b374
Diff Diff

Regressions in System.Numerics.Tests.Perf_Matrix4x4

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
CreateScaleFromVectorWithCenterBenchmark - Duration of single invocation 48.72 ns 71.12 ns 1.46 0.24 False
CreateScaleFromScalarWithCenterBenchmark - Duration of single invocation 31.06 ns 65.34 ns 2.10 0.33 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Matrix4x4*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Matrix4x4.CreateScaleFromVectorWithCenterBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 71.12071711448095 > 51.37175104774888.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 12/5/2022 3:14:07 PM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -30.042565700927085 (T) = (0 -74.16010821325703) / Math.Sqrt((6.582288291816523 / (20)) + (10.828568657684224 / (38))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (38) - 2, .025) and -0.46509830556410764 = (50.617837677931874 - 74.16010821325703) / 50.617837677931874 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Matrix4x4.CreateScaleFromScalarWithCenterBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 65.3419631014328 > 32.34757438260667.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 10/19/2022 7:27:15 PM, 12/5/2022 3:14:07 PM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -66.88073059266382 (T) = (0 -62.25724408227327) / Math.Sqrt((0.41509577176642365 / (20)) + (7.453106568499127 / (38))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (38) - 2, .025) and -1.0011986400652368 = (31.109977208581228 - 62.25724408227327) / 31.109977208581228 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler performanceautofiler bot added refs/heads/main untriaged New issue has not been triaged by the area owner labels Dec 13, 2022
@kotlarmilos
Copy link
Member

Regression caused by #79172

@kotlarmilos kotlarmilos removed the untriaged New issue has not been triaged by the area owner label Dec 13, 2022
@kotlarmilos kotlarmilos self-assigned this Dec 13, 2022
@kotlarmilos kotlarmilos transferred this issue from dotnet/perf-autofiling-issues Dec 13, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@ghost ghost added the untriaged New issue has not been triaged by the area owner label Dec 13, 2022
@ghost
Copy link

ghost commented Dec 13, 2022

Tagging subscribers to this area: @BrzVlad
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 315f4ab59adf7eaec815885825c5b0d65fc3ab99
Compare 21a3647f09ca92cbf890129d4e917cbd01a1b374
Diff Diff

Regressions in System.Numerics.Tests.Perf_Matrix4x4

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
CreateScaleFromVectorWithCenterBenchmark - Duration of single invocation 48.72 ns 71.12 ns 1.46 0.24 False
CreateScaleFromScalarWithCenterBenchmark - Duration of single invocation 31.06 ns 65.34 ns 2.10 0.33 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Matrix4x4*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Matrix4x4.CreateScaleFromVectorWithCenterBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 71.12071711448095 > 51.37175104774888.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 12/5/2022 3:14:07 PM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -30.042565700927085 (T) = (0 -74.16010821325703) / Math.Sqrt((6.582288291816523 / (20)) + (10.828568657684224 / (38))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (38) - 2, .025) and -0.46509830556410764 = (50.617837677931874 - 74.16010821325703) / 50.617837677931874 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Matrix4x4.CreateScaleFromScalarWithCenterBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 65.3419631014328 > 32.34757438260667.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 10/19/2022 7:27:15 PM, 12/5/2022 3:14:07 PM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -66.88073059266382 (T) = (0 -62.25724408227327) / Math.Sqrt((0.41509577176642365 / (20)) + (7.453106568499127 / (38))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (38) - 2, .025) and -1.0011986400652368 = (31.109977208581228 - 62.25724408227327) / 31.109977208581228 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: kotlarmilos
Labels:

untriaged, area-Codegen-Interpreter-mono, refs/heads/main, ubuntu 18.04, Regression, x64

Milestone: -

@kotlarmilos kotlarmilos removed the untriaged New issue has not been triaged by the area owner label Dec 13, 2022
@kotlarmilos kotlarmilos added this to the 8.0.0 milestone Dec 13, 2022
@jeffhandley jeffhandley added os-linux Linux OS (any supported distro) arch-x64 and removed refs/heads/main labels Dec 28, 2022
@kotlarmilos
Copy link
Member

Similar to dotnet/perf-autofiling-issues#11453.

@ghost ghost locked as resolved and limited conversation to collaborators Mar 3, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

2 participants