Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Perf] Linux/arm64: 45 Improvements on 2/3/2023 1:33:16 AM #12652

Closed
performanceautofiler bot opened this issue Feb 7, 2023 · 1 comment
Closed

[Perf] Linux/arm64: 45 Improvements on 2/3/2023 1:33:16 AM #12652

performanceautofiler bot opened this issue Feb 7, 2023 · 1 comment
Labels

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Feb 7, 2023

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 00c60e41ac459b005338329cd9b5fd5d6652777c
Compare efe93b7d30add5ae40a57b92140990a522e7ee9c
Diff Diff

Improvements in System.Numerics.Tests.Perf_Quaternion

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
MultiplyByScalarBenchmark - Duration of single invocation 65.26 ns 47.62 ns 0.73 0.17 False
NegateBenchmark - Duration of single invocation 63.03 ns 58.16 ns 0.92 0.13 False
ConcatenateBenchmark - Duration of single invocation 157.42 ns 132.83 ns 0.84 0.11 False
MultiplyByScalarOperatorBenchmark - Duration of single invocation 65.78 ns 45.53 ns 0.69 0.19 False
DivisionOperatorBenchmark - Duration of single invocation 197.49 ns 169.76 ns 0.86 0.06 False
SubtractionOperatorBenchmark - Duration of single invocation 102.08 ns 69.48 ns 0.68 0.08 False
InequalityOperatorBenchmark - Duration of single invocation 108.28 ns 64.54 ns 0.60 0.04 False
LengthSquaredBenchmark - Duration of single invocation 82.82 ns 49.67 ns 0.60 0.10 False
SubtractBenchmark - Duration of single invocation 100.97 ns 70.01 ns 0.69 0.19 False
CreateFromVector3WithScalarBenchmark - Duration of single invocation 49.71 ns 35.14 ns 0.71 0.22 False
AddOperatorBenchmark - Duration of single invocation 102.06 ns 68.39 ns 0.67 0.12 False
EqualityOperatorBenchmark - Duration of single invocation 93.46 ns 64.18 ns 0.69 0.12 False
DivideBenchmark - Duration of single invocation 102.42 ns 71.07 ns 0.69 0.09 False
DotBenchmark - Duration of single invocation 98.94 ns 63.45 ns 0.64 0.10 False
AddBenchmark - Duration of single invocation 102.42 ns 69.56 ns 0.68 0.07 False
MultiplyByQuaternionBenchmark - Duration of single invocation 157.59 ns 142.79 ns 0.91 0.14 False
LengthBenchmark - Duration of single invocation 97.77 ns 50.80 ns 0.52 0.33 False
NormalizeBenchmark - Duration of single invocation 102.64 ns 73.67 ns 0.72 0.06 False
IsIdentityBenchmark - Duration of single invocation 114.43 ns 72.41 ns 0.63 0.22 False
MultiplyByQuaternionOperatorBenchmark - Duration of single invocation 156.63 ns 132.10 ns 0.84 0.17 False
EqualsBenchmark - Duration of single invocation 143.35 ns 113.55 ns 0.79 0.11 False
IdentityBenchmark - Duration of single invocation 34.27 ns 17.11 ns 0.50 0.09 False
CreateFromScalarXYZWBenchmark - Duration of single invocation 25.47 ns 18.94 ns 0.74 0.27 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Quaternion*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Quaternion.MultiplyByScalarBenchmark


Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 32.87744567428963 (T) = (0 -46.43179156542937) / Math.Sqrt((9.562971844310212 / (85)) + (1.1901138274353256 / (3))) is greater than 1.9879342062385923 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (85) + (3) - 2, .975) and 0.33567128069189056 = (69.89279586435393 - 46.43179156542937) / 69.89279586435393 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.NegateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 10/7/2022 12:48:42 AM, 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 12.911959056456912 (T) = (0 -56.806994119607424) / Math.Sqrt((7.252243494916296 / (85)) + (1.384159176632874 / (3))) is greater than 1.9879342062385923 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (85) + (3) - 2, .975) and 0.1438805171653802 = (66.35404900670981 - 56.806994119607424) / 66.35404900670981 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.ConcatenateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 10/7/2022 12:48:42 AM, 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 28.72447687905941 (T) = (0 -133.6125595840923) / Math.Sqrt((19.913711850505617 / (79)) + (0.4565374152202734 / (3))) is greater than 1.990063421253899 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (79) + (3) - 2, .975) and 0.1202511748114275 = (151.8758033640485 - 133.6125595840923) / 151.8758033640485 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.MultiplyByScalarOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 36.1095506290876 (T) = (0 -45.86304352145689) / Math.Sqrt((4.091973999864425 / (81)) + (0.6408353772488902 / (3))) is greater than 1.9893185571361351 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (81) + (3) - 2, .975) and 0.2880734619355271 = (64.42103372933047 - 45.86304352145689) / 64.42103372933047 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.DivisionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 10/7/2022 12:48:42 AM, 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 23.62315878305684 (T) = (0 -170.93079314261524) / Math.Sqrt((26.057632623925798 / (85)) + (1.0705058041238265 / (3))) is greater than 1.9879342062385923 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (85) + (3) - 2, .975) and 0.10117623583785015 = (190.1716442732804 - 170.93079314261524) / 190.1716442732804 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 27.90989457522394 (T) = (0 -67.26289182840017) / Math.Sqrt((8.684191202800978 / (85)) + (4.006841409154301 / (3))) is greater than 1.9879342062385923 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (85) + (3) - 2, .975) and 0.332238711166779 = (100.72894753442296 - 67.26289182840017) / 100.72894753442296 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.InequalityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 73.26585539964174 (T) = (0 -65.34132551310803) / Math.Sqrt((5.288012760521424 / (85)) + (0.7209883915739761 / (3))) is greater than 1.9879342062385923 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (85) + (3) - 2, .975) and 0.3814735066102295 = (105.64030192952876 - 65.34132551310803) / 105.64030192952876 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.LengthSquaredBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 95.7924488928885 (T) = (0 -49.49064334323962) / Math.Sqrt((5.852468111397341 / (84)) + (0.037593246767559346 / (3))) is greater than 1.9882679074768839 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (84) + (3) - 2, .975) and 0.3568919783657613 = (76.95541289856112 - 49.49064334323962) / 76.95541289856112 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 37.83884691555698 (T) = (0 -68.13377119690405) / Math.Sqrt((16.733549845942424 / (85)) + (2.7718479615609875 / (3))) is greater than 1.9879342062385923 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (85) + (3) - 2, .975) and 0.3702583039430642 = (108.1932030601067 - 68.13377119690405) / 108.1932030601067 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.CreateFromVector3WithScalarBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 13.120139786695548 (T) = (0 -35.53115856632288) / Math.Sqrt((3.876270664925651 / (85)) + (2.536965990151565 / (3))) is greater than 1.9879342062385923 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (85) + (3) - 2, .975) and 0.25849185587366225 = (47.917421875637686 - 35.53115856632288) / 47.917421875637686 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.AddOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 33.583182680361325 (T) = (0 -67.5004191325602) / Math.Sqrt((9.53758763812192 / (85)) + (2.6144088678236637 / (3))) is greater than 1.9879342062385923 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (85) + (3) - 2, .975) and 0.33040863195537956 = (100.80837709972104 - 67.5004191325602) / 100.80837709972104 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.EqualityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 22.341967923549962 (T) = (0 -64.82706868545114) / Math.Sqrt((5.25213709854781 / (85)) + (3.6702976384229533 / (3))) is greater than 1.9879342062385923 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (85) + (3) - 2, .975) and 0.28094266722436806 = (90.15563256300062 - 64.82706868545114) / 90.15563256300062 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.DivideBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 26.66178015516263 (T) = (0 -68.57579585349053) / Math.Sqrt((9.06711225256664 / (85)) + (6.19795387794859 / (3))) is greater than 1.9879342062385923 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (85) + (3) - 2, .975) and 0.36430366063897296 = (107.87508375841804 - 68.57579585349053) / 107.87508375841804 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.DotBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 26.203063871751805 (T) = (0 -61.95597212398676) / Math.Sqrt((6.324563286348814 / (85)) + (4.668067103863303 / (3))) is greater than 1.9879342062385923 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (85) + (3) - 2, .975) and 0.35066302675229327 = (95.41420660848773 - 61.95597212398676) / 95.41420660848773 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 20.656770385250642 (T) = (0 -69.08662947489957) / Math.Sqrt((9.360900005165467 / (81)) + (10.188316833363261 / (3))) is greater than 1.9893185571361351 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (81) + (3) - 2, .975) and 0.3591001445594652 = (107.79629436397914 - 69.08662947489957) / 107.79629436397914 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.MultiplyByQuaternionBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 10/7/2022 12:48:42 AM, 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 7.592580857117215 (T) = (0 -136.94193904469952) / Math.Sqrt((22.87425083113539 / (85)) + (25.798267252802894 / (3))) is greater than 1.9879342062385923 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (85) + (3) - 2, .975) and 0.14171346086998182 = (159.55270507155743 - 136.94193904469952) / 159.55270507155743 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.LengthBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 92.47684295934174 (T) = (0 -50.730537788364416) / Math.Sqrt((18.053148426475747 / (85)) + (0.008958513576038548 / (3))) is greater than 1.9879342062385923 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (85) + (3) - 2, .975) and 0.4582836022335257 = (93.6477795346959 - 50.730537788364416) / 93.6477795346959 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.NormalizeBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 48.906681986679004 (T) = (0 -74.2276953909225) / Math.Sqrt((8.718330898434367 / (85)) + (0.4483262914421198 / (3))) is greater than 1.9879342062385923 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (85) + (3) - 2, .975) and 0.24854915319002055 = (98.77917591819892 - 74.2276953909225) / 98.77917591819892 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.IsIdentityBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 24.06905649262033 (T) = (0 -69.0813912507864) / Math.Sqrt((13.66882116043021 / (85)) + (8.397580004850555 / (3))) is greater than 1.9879342062385923 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (85) + (3) - 2, .975) and 0.3747802813271358 = (110.49138276928224 - 69.0813912507864) / 110.49138276928224 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.MultiplyByQuaternionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 38.73419908131432 (T) = (0 -131.85396727919115) / Math.Sqrt((18.320476242189354 / (82)) + (0.1011513685863208 / (3))) is greater than 1.9889597801747605 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (82) + (3) - 2, .975) and 0.1296514432591626 = (151.49558904646193 - 131.85396727919115) / 151.49558904646193 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.EqualsBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 45.507652547299244 (T) = (0 -112.41056921663501) / Math.Sqrt((10.705459979497235 / (85)) + (1.205825532645214 / (3))) is greater than 1.9879342062385923 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (85) + (3) - 2, .975) and 0.22728375743320842 = (145.47457789062656 - 112.41056921663501) / 145.47457789062656 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.IdentityBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 32.951154139862446 (T) = (0 -17.625839256145834) / Math.Sqrt((0.5391088251042022 / (85)) + (0.6807610609150401 / (3))) is greater than 1.9879342062385923 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (85) + (3) - 2, .975) and 0.47448856357866814 = (33.540353329274105 - 17.625839256145834) / 33.540353329274105 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.CreateFromScalarXYZWBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 38.35487788210764 (T) = (0 -19.035345747978084) / Math.Sqrt((1.4102268963991516 / (84)) + (0.0112765449791293 / (3))) is greater than 1.9882679074768839 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (84) + (3) - 2, .975) and 0.22410029150866592 = (24.53325544481847 - 19.035345747978084) / 24.53325544481847 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline 00c60e41ac459b005338329cd9b5fd5d6652777c
Compare efe93b7d30add5ae40a57b92140990a522e7ee9c
Diff Diff

Improvements in System.Numerics.Tests.Perf_Vector4

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TransformVector3ByQuaternionBenchmark - Duration of single invocation 173.15 ns 154.59 ns 0.89 0.08 False
TransformByQuaternionBenchmark - Duration of single invocation 173.51 ns 155.84 ns 0.90 0.04 False
UnitXBenchmark - Duration of single invocation 28.51 ns 20.04 ns 0.70 0.28 False
UnitYBenchmark - Duration of single invocation 27.85 ns 18.55 ns 0.67 0.24 False
UnitWBenchmark - Duration of single invocation 27.78 ns 19.90 ns 0.72 0.27 False
TransformVector2ByQuaternionBenchmark - Duration of single invocation 147.37 ns 128.76 ns 0.87 0.11 False
UnitZBenchmark - Duration of single invocation 27.41 ns 17.23 ns 0.63 0.27 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector4*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Vector4.TransformVector3ByQuaternionBenchmark


Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 12.161647561956338 (T) = (0 -153.01449136418378) / Math.Sqrt((13.4585968436967 / (82)) + (2.7309043089995835 / (3))) is greater than 1.9889597801747605 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (82) + (3) - 2, .975) and 0.07611446771364316 = (165.62061642584223 - 153.01449136418378) / 165.62061642584223 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector4.TransformByQuaternionBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 11.727136578724902 (T) = (0 -154.0335570618055) / Math.Sqrt((14.423684942927434 / (85)) + (2.464835895579567 / (3))) is greater than 1.9879342062385923 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (85) + (3) - 2, .975) and 0.0704607776349196 = (165.70958315227304 - 154.0335570618055) / 165.70958315227304 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector4.UnitXBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 10/7/2022 12:48:42 AM, 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 15.997375029128893 (T) = (0 -19.239305653766166) / Math.Sqrt((3.920497402553979 / (85)) + (0.5485506463364255 / (3))) is greater than 1.9879342062385923 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (85) + (3) - 2, .975) and 0.28463103009078644 = (26.89424124198146 - 19.239305653766166) / 26.89424124198146 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector4.UnitYBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 10/7/2022 12:48:42 AM, 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 32.55961488503379 (T) = (0 -18.654736484368865) / Math.Sqrt((3.1548814462909425 / (84)) + (0.07540649759931446 / (3))) is greater than 1.9882679074768839 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (84) + (3) - 2, .975) and 0.30411569156654905 = (26.8072382985093 - 18.654736484368865) / 26.8072382985093 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector4.UnitWBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 10/7/2022 12:48:42 AM, 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 23.10133271069257 (T) = (0 -19.40188947565857) / Math.Sqrt((3.5533918904433492 / (85)) + (0.18611010109949394 / (3))) is greater than 1.9879342062385923 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (85) + (3) - 2, .975) and 0.2772936665421306 = (26.84615946677547 - 19.40188947565857) / 26.84615946677547 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector4.TransformVector2ByQuaternionBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 18.792984442153525 (T) = (0 -127.86983427857142) / Math.Sqrt((14.027347711344513 / (85)) + (1.0646384263391526 / (3))) is greater than 1.9879342062385923 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (85) + (3) - 2, .975) and 0.09581785629980188 = (141.42043743010427 - 127.86983427857142) / 141.42043743010427 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector4.UnitZBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 10/7/2022 12:48:42 AM, 2/2/2023 2:17:49 AM, 2/7/2023 1:28:31 AM falls between 1/29/2023 12:56:15 AM and 2/7/2023 1:28:31 AM.
IsImprovementStdDev: Marked as improvement because 10.67409301045115 (T) = (0 -18.713328673174132) / Math.Sqrt((3.648296583854809 / (85)) + (1.658778729787623 / (3))) is greater than 1.9879342062385923 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (85) + (3) - 2, .975) and 0.30569959961385834 = (26.95278392863743 - 18.713328673174132) / 26.95278392863743 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler performanceautofiler bot added ampere arm64 PGO Applied if there were any profile guided optimization updates in the observed interval. untriaged labels Feb 7, 2023
@kotlarmilos
Copy link
Member

Duplicate of #12504

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant