Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/arm64: 45 Improvements on 12/16/2022 10:05:39 PM #10938

Closed
performanceautofiler bot opened this issue Dec 20, 2022 · 2 comments
Closed

[Perf] Linux/arm64: 45 Improvements on 12/16/2022 10:05:39 PM #10938

performanceautofiler bot opened this issue Dec 20, 2022 · 2 comments
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-improvement runtime-mono untriaged

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Dec 20, 2022 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline caa0bd5b30705a2d3f9b67031458c4e468fff4c1
Compare 4dd8a7801fcc2fa888bdde10d2d3ebd1bfe027ac
Diff Diff

Improvements in System.Tests.Perf_Array

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ArrayRetrieve3D - Duration of single invocation 225.56 ms 82.97 ms 0.37 0.18 False
ArrayAssign3D - Duration of single invocation 202.70 ms 75.75 ms 0.37 0.18 False
Clear - Duration of single invocation 1.48 μs 1.02 μs 0.69 0.14 False
ArrayRetrieve1D - Duration of single invocation 103.53 ms 48.76 ms 0.47 0.20 False
ArrayAssign2D - Duration of single invocation 160.54 ms 69.58 ms 0.43 0.25 False
ArrayAssign1D - Duration of single invocation 95.15 ms 47.94 ms 0.50 0.23 False
ArrayRetrieve2D - Duration of single invocation 162.51 ms 66.27 ms 0.41 0.11 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Array*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Array.ArrayRetrieve3D

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 5.0339293304978545 (T) = (0 -81335357.37916666) / Math.Sqrt((3971933629914975.5 / (30)) + (5373766385504.952 / (2))) is greater than 2.0422724562973102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (2) - 2, .975) and 0.4183795439347742 = (139842669.7874693 - 81335357.37916666) / 139842669.7874693 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayAssign3D

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 4.66191044099335 (T) = (0 -77318197.02619047) / Math.Sqrt((2509639025271627.5 / (40)) + (4926312017372.142 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.32744966934850256 = (114962692.75682695 - 77318197.02619047) / 114962692.75682695 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.Clear

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 2.3233188428441744 (T) = (0 -998.8936380157048) / Math.Sqrt((43666.01785968943 / (40)) + (922.4340130420228 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.08395978209410425 = (1090.447360814807 - 998.8936380157048) / 1090.447360814807 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayRetrieve1D

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 9.690127242539504 (T) = (0 -47990895.55064103) / Math.Sqrt((298374622322319 / (40)) + (1177713691557.6255 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.36420063568800964 = (75481194.6101469 - 47990895.55064103) / 75481194.6101469 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayAssign2D

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 5.193741286851943 (T) = (0 -67475093.775) / Math.Sqrt((1247159076722226.8 / (40)) + (8863728957347.43 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.3147555814862734 = (98468651.40667814 - 67475093.775) / 98468651.40667814 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayAssign1D

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 9.202894806315724 (T) = (0 -47874663.56666666) / Math.Sqrt((310575071815689.25 / (40)) + (9360367001.63467 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.348873638515262 = (73525918.16049336 - 47874663.56666666) / 73525918.16049336 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayRetrieve2D

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 5.863886821557396 (T) = (0 -65220824.4672619) / Math.Sqrt((1444715140003599 / (33)) + (2210707075553.169 / (2))) is greater than 2.0345152974461924 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (2) - 2, .975) and 0.37591611514037215 = (104506503.1312765 - 65220824.4672619) / 104506503.1312765 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline caa0bd5b30705a2d3f9b67031458c4e468fff4c1
Compare 4dd8a7801fcc2fa888bdde10d2d3ebd1bfe027ac
Diff Diff

Improvements in System.Collections.CtorFromCollection<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableList - Duration of single invocation 372.96 μs 207.71 μs 0.56 0.07 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollection&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorFromCollection<String>.ImmutableList(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 2.710349037133126 (T) = (0 -210267.99415747984) / Math.Sqrt((4887689139.203124 / (39)) + (13096315.20526544 / (2))) is greater than 2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (2) - 2, .975) and 0.12893773856699164 = (241392.61160455077 - 210267.99415747984) / 241392.61160455077 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline caa0bd5b30705a2d3f9b67031458c4e468fff4c1
Compare 4dd8a7801fcc2fa888bdde10d2d3ebd1bfe027ac
Diff Diff

Improvements in MicroBenchmarks.Serializers.Binary_FromStream<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BinaryFormatter_ - Duration of single invocation 468.70 μs 398.84 μs 0.85 0.07 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Binary_FromStream&lt;IndexViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Binary_FromStream<IndexViewModel>.BinaryFormatter_

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 22.486338434986916 (T) = (0 -397570.7359895833) / Math.Sqrt((324396119.7840459 / (40)) + (3209639.1688511204 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.1498671160244694 = (467657.16687772144 - 397570.7359895833) / 467657.16687772144 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline caa0bd5b30705a2d3f9b67031458c4e468fff4c1
Compare 4dd8a7801fcc2fa888bdde10d2d3ebd1bfe027ac
Diff Diff

Improvements in System.Collections.CtorFromCollectionNonGeneric<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Stack - Duration of single invocation 666.67 μs 482.90 μs 0.72 0.18 False
Queue - Duration of single invocation 655.52 μs 477.30 μs 0.73 0.16 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollectionNonGeneric&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorFromCollectionNonGeneric<String>.Stack(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 29.468270815275392 (T) = (0 -480484.81487193366) / Math.Sqrt((2506330718.985884 / (40)) + (11688893.060756609 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.336697818103848 = (724382.9855924692 - 480484.81487193366) / 724382.9855924692 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollectionNonGeneric&lt;String&gt;.Queue(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 28.028171342767696 (T) = (0 -479001.775363005) / Math.Sqrt((3114718450.440667 / (40)) + (5810943.418562875 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.3446432038231761 = (730902.278205358 - 479001.775363005) / 730902.278205358 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline caa0bd5b30705a2d3f9b67031458c4e468fff4c1
Compare 4dd8a7801fcc2fa888bdde10d2d3ebd1bfe027ac
Diff Diff

Improvements in Benchstone.MDBenchI.MDGeneralArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test2 - Duration of single invocation 7.72 secs 1.55 secs 0.20 0.16 False
Test - Duration of single invocation 6.84 secs 1.49 secs 0.22 0.11 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.MDBenchI.MDGeneralArray*'

Payloads

Baseline
Compare

Histogram

Benchstone.MDBenchI.MDGeneralArray.Test2

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 2.9191661638077817 (T) = (0 -1515651441.2785716) / Math.Sqrt((6.329366814144505E+18 / (40)) + (2303213437364393 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.43468506614942803 = (2681074478.1760874 - 1515651441.2785716) / 2681074478.1760874 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### Benchstone.MDBenchI.MDGeneralArray.Test

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 2.986453830682803 (T) = (0 -1480388244.8805556) / Math.Sqrt((6.233020722850748E+18 / (40)) + (322960312916357.06 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.4434408537560305 = (2659893840.342393 - 1480388244.8805556) / 2659893840.342393 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline caa0bd5b30705a2d3f9b67031458c4e468fff4c1
Compare 4dd8a7801fcc2fa888bdde10d2d3ebd1bfe027ac
Diff Diff

Improvements in System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
CreateFromFile - Duration of single invocation 31.23 μs 27.97 μs 0.90 0.05 False
CreateFromFile - Duration of single invocation 30.67 μs 28.10 μs 0.92 0.06 False
CreateFromFile - Duration of single invocation 31.00 μs 28.32 μs 0.91 0.02 False
CreateFromFile_Read - Duration of single invocation 24.41 μs 21.95 μs 0.90 0.02 False
CreateFromFile_Read - Duration of single invocation 24.22 μs 22.14 μs 0.91 0.02 False
CreateFromFile_Read - Duration of single invocation 24.25 μs 21.98 μs 0.91 0.02 False
CreateFromFile - Duration of single invocation 31.15 μs 27.79 μs 0.89 0.09 False
CreateFromFile_Read - Duration of single invocation 24.35 μs 22.62 μs 0.93 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile*'

Payloads

Baseline
Compare

Histogram

System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateFromFile(capacity: 10000000)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 2.1618650410912266 (T) = (0 -28793.877895538524) / Math.Sqrt((150627.3472743609 / (40)) + (1371279.5397918308 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.05868164815785234 = (30588.884025462037 - 28793.877895538524) / 30588.884025462037 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateFromFile(capacity: 10000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 24.246655643076114 (T) = (0 -28090.03975311177) / Math.Sqrt((287021.31752654427 / (29)) + (332.4147383267336 / (2))) is greater than 2.045229642128227 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (2) - 2, .975) and 0.07969071007856406 = (30522.390744865494 - 28090.03975311177) / 30522.390744865494 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateFromFile(capacity: 1000000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 31.17459636327631 (T) = (0 -28300.55205904513) / Math.Sqrt((192115.3417110378 / (40)) + (645.0809893986457 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.07309786848937998 = (30532.405846259375 - 28300.55205904513) / 30532.405846259375 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateFromFile_Read(capacity: 100000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 46.175415908023766 (T) = (0 -21958.881130166556) / Math.Sqrt((93168.47001280941 / (40)) + (113.96958950238881 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.09315122783061686 = (24214.490667099926 - 21958.881130166556) / 24214.490667099926 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateFromFile_Read(capacity: 1000000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 20.84537199369859 (T) = (0 -22042.708831504908) / Math.Sqrt((51409.20256788578 / (40)) + (19234.802000913085 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.08986981770613361 = (24219.292207131388 - 22042.708831504908) / 24219.292207131388 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateFromFile_Read(capacity: 10000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 21.452563453631296 (T) = (0 -22073.543244453398) / Math.Sqrt((94800.88493086606 / (40)) + (15879.53273440986 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.08981741337665126 = (24251.774939293427 - 22073.543244453398) / 24251.774939293427 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateFromFile(capacity: 100000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 17.336481838535516 (T) = (0 -27910.945760346665) / Math.Sqrt((304237.4813461172 / (29)) + (30041.24235366686 / (2))) is greater than 2.045229642128227 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (2) - 2, .975) and 0.0902556439481722 = (30679.987817101213 - 27910.945760346665) / 30679.987817101213 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateFromFile_Read(capacity: 10000000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 5.7753745737258315 (T) = (0 -22255.84034325145) / Math.Sqrt((88944.6570673915 / (39)) + (267319.1995401777 / (2))) is greater than 2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (2) - 2, .975) and 0.0873227470089238 = (24385.22519358665 - 22255.84034325145) / 24385.22519358665 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline caa0bd5b30705a2d3f9b67031458c4e468fff4c1
Compare 4dd8a7801fcc2fa888bdde10d2d3ebd1bfe027ac
Diff Diff

Improvements in System.Collections.CreateAddAndClear<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ICollection - Duration of single invocation 40.27 μs 29.60 μs 0.74 0.16 False
ImmutableArray - Duration of single invocation 1.25 ms 431.40 μs 0.35 0.07 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndClear&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndClear<Int32>.ICollection(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 2.1285209293134235 (T) = (0 -28897.480799749086) / Math.Sqrt((27084248.287030872 / (40)) + (976135.9998696437 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.07365247065216206 = (31195.07515726126 - 28897.480799749086) / 31195.07515726126 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.ImmutableArray(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 2.7607446864158796 (T) = (0 -432616.4431603478) / Math.Sqrt((140457528037.6452 / (40)) + (2950712.1840050346 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.27443214860853204 = (596245.3302343696 - 432616.4431603478) / 596245.3302343696 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline caa0bd5b30705a2d3f9b67031458c4e468fff4c1
Compare 4dd8a7801fcc2fa888bdde10d2d3ebd1bfe027ac
Diff Diff

Improvements in System.Linq.Tests.Perf_Enumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToArray - Duration of single invocation 1.74 μs 541.05 ns 0.31 0.11 False
SelectToArray - Duration of single invocation 22.07 μs 14.98 μs 0.68 0.11 False
ToList - Duration of single invocation 1.79 μs 576.07 ns 0.32 0.07 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Linq.Tests.Perf_Enumerable*'

Payloads

Baseline
Compare

Histogram

System.Linq.Tests.Perf_Enumerable.ToArray(input: ICollection)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 2.6975686839095414 (T) = (0 -539.1655455202495) / Math.Sqrt((291000.4596675854 / (40)) + (7.138791472669899 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.2991547342186836 = (769.3075374051031 - 539.1655455202495) / 769.3075374051031 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SelectToArray(input: IEnumerable)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 2.4457326851220693 (T) = (0 -14870.390889112896) / Math.Sqrt((10130228.578039808 / (40)) + (24457.915152041318 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.07812306572500702 = (16130.559661748848 - 14870.390889112896) / 16130.559661748848 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.ToList(input: ICollection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 2.719326320274447 (T) = (0 -572.1250848815775) / Math.Sqrt((283962.5817102887 / (40)) + (31.049502674314436 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.28617750055024394 = (801.4948888870766 - 572.1250848815775) / 801.4948888870766 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline caa0bd5b30705a2d3f9b67031458c4e468fff4c1
Compare 4dd8a7801fcc2fa888bdde10d2d3ebd1bfe027ac
Diff Diff

Improvements in System.Security.Cryptography.Tests.Perf_CryptoConfig

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
CreateFromName - Duration of single invocation 12.01 μs 9.76 μs 0.81 0.09 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Security.Cryptography.Tests.Perf_CryptoConfig*'

Payloads

Baseline
Compare

Histogram

System.Security.Cryptography.Tests.Perf_CryptoConfig.CreateFromName(name: "SHA512")

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 4.154458743083619 (T) = (0 -9808.877747379956) / Math.Sqrt((833267.5397476993 / (40)) + (4954.431622840478 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.060735678020454986 = (10443.149513767617 - 9808.877747379956) / 10443.149513767617 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline caa0bd5b30705a2d3f9b67031458c4e468fff4c1
Compare 4dd8a7801fcc2fa888bdde10d2d3ebd1bfe027ac
Diff Diff

Improvements in System.Collections.CtorFromCollectionNonGeneric<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Stack - Duration of single invocation 741.15 μs 558.66 μs 0.75 0.14 False
ArrayList - Duration of single invocation 584.74 μs 178.30 μs 0.30 0.16 False
Queue - Duration of single invocation 753.63 μs 602.30 μs 0.80 0.25 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollectionNonGeneric&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorFromCollectionNonGeneric<Int32>.Stack(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 21.703737700648723 (T) = (0 -550711.5226766767) / Math.Sqrt((2962676092.871006 / (39)) + (126288823.10178912 / (2))) is greater than 2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (2) - 2, .975) and 0.31732499026444566 = (806696.4731725043 - 550711.5226766767) / 806696.4731725043 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollectionNonGeneric&lt;Int32&gt;.ArrayList(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 3.203147500270608 (T) = (0 -178519.66124695615) / Math.Sqrt((32554177740.47427 / (40)) + (93039.85375229472 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.33857622921087277 = (269902.0947402133 - 178519.66124695615) / 269902.0947402133 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollectionNonGeneric&lt;Int32&gt;.Queue(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 7.33086882497782 (T) = (0 -571037.7366462201) / Math.Sqrt((2751083034.959003 / (40)) + (1954610825.896625 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.2933938558609675 = (808141.4821859547 - 571037.7366462201) / 808141.4821859547 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline caa0bd5b30705a2d3f9b67031458c4e468fff4c1
Compare 4dd8a7801fcc2fa888bdde10d2d3ebd1bfe027ac
Diff Diff

Improvements in System.Collections.CreateAddAndRemove<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
List - Duration of single invocation 597.76 μs 222.30 μs 0.37 0.14 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndRemove&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndRemove<Int32>.List(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 2.6566313584532817 (T) = (0 -221302.00084884305) / Math.Sqrt((30180043872.98944 / (40)) + (1985579.2829398513 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.24809774573868004 = (294322.831983332 - 221302.00084884305) / 294322.831983332 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline caa0bd5b30705a2d3f9b67031458c4e468fff4c1
Compare 4dd8a7801fcc2fa888bdde10d2d3ebd1bfe027ac
Diff Diff

Improvements in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitArrayBitArrayCtor - Duration of single invocation 2.18 μs 571.29 ns 0.26 0.05 False
BitArrayIntArrayCtor - Duration of single invocation 2.20 μs 570.86 ns 0.26 0.09 False
BitArrayBitArrayCtor - Duration of single invocation 2.32 μs 575.36 ns 0.25 0.09 False
BitArrayIntArrayCtor - Duration of single invocation 2.57 μs 941.68 ns 0.37 0.09 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_BitArray.BitArrayBitArrayCtor(Size: 4)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 2.8539333428787326 (T) = (0 -566.382162249044) / Math.Sqrt((541398.2358036245 / (40)) + (48.25955479916936 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.36977904696343517 = (898.704112454641 - 566.382162249044) / 898.704112454641 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayIntArrayCtor(Size: 4)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 2.823656991512407 (T) = (0 -568.5070291714162) / Math.Sqrt((547939.0474538844 / (40)) + (11.116972480615694 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.36766246062872576 = (899.0562694359029 - 568.5070291714162) / 899.0562694359029 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayBitArrayCtor(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 2.897065275382179 (T) = (0 -573.9311808422659) / Math.Sqrt((568223.2408169698 / (39)) + (4.068934973066978 / (2))) is greater than 2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (2) - 2, .975) and 0.37862525067261255 = (923.6474148064799 - 573.9311808422659) / 923.6474148064799 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayIntArrayCtor(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 2.7595238540784153 (T) = (0 -941.6414962250096) / Math.Sqrt((558714.4381210421 / (40)) + (0.0023918266978487993 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.25725035681614433 = (1267.7777833572402 - 941.6414962250096) / 1267.7777833572402 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline caa0bd5b30705a2d3f9b67031458c4e468fff4c1
Compare 4dd8a7801fcc2fa888bdde10d2d3ebd1bfe027ac
Diff Diff

Improvements in System.Collections.AddGivenSize<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedList - Duration of single invocation 1.32 ms 641.31 μs 0.48 0.07 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.AddGivenSize&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.AddGivenSize<Int32>.SortedList(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 2.627128609354501 (T) = (0 -639571.6911666666) / Math.Sqrt((103111053235.13097 / (40)) + (6054008.738596458 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.1726474494901628 = (773034.0479068386 - 639571.6911666666) / 773034.0479068386 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline caa0bd5b30705a2d3f9b67031458c4e468fff4c1
Compare 4dd8a7801fcc2fa888bdde10d2d3ebd1bfe027ac
Diff Diff

Improvements in System.Runtime.Serialization.Formatters.Tests.Perf_BinaryFormatter

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeLargeList - Duration of single invocation 2.08 secs 1.87 secs 0.90 0.09 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Runtime.Serialization.Formatters.Tests.Perf_BinaryFormatter*'

Payloads

Baseline
Compare

Histogram

System.Runtime.Serialization.Formatters.Tests.Perf_BinaryFormatter.DeserializeLargeList

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 9.797271440302875 (T) = (0 -1839352072.4666667) / Math.Sqrt((7734884659865259 / (40)) + (2518493579723668 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.16875129120917326 = (2212757810.0449314 - 1839352072.4666667) / 2212757810.0449314 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline caa0bd5b30705a2d3f9b67031458c4e468fff4c1
Compare 4dd8a7801fcc2fa888bdde10d2d3ebd1bfe027ac
Diff Diff

Improvements in System.Collections.CtorFromCollection<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableList - Duration of single invocation 386.04 μs 215.79 μs 0.56 0.12 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollection&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorFromCollection<Int32>.ImmutableList(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 2.5992495162777347 (T) = (0 -216216.01006944448) / Math.Sqrt((5211372445.411461 / (39)) + (360397.0607721177 / (2))) is greater than 2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (39) + (2) - 2, .975) and 0.12208171077831982 = (246282.61277154976 - 216216.01006944448) / 246282.61277154976 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline caa0bd5b30705a2d3f9b67031458c4e468fff4c1
Compare 4dd8a7801fcc2fa888bdde10d2d3ebd1bfe027ac
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.ReadJson<HashSet<String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromStream - Duration of single invocation 222.09 μs 207.72 μs 0.94 0.00 False
DeserializeFromUtf8Bytes - Duration of single invocation 199.37 μs 186.42 μs 0.94 0.00 False
DeserializeFromUtf8Bytes - Duration of single invocation 199.90 μs 186.79 μs 0.93 0.04 False
DeserializeFromString - Duration of single invocation 239.36 μs 218.16 μs 0.91 0.00 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<HashSet<String>>.DeserializeFromStream(Mode: Reflection)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 14.608088789835158 (T) = (0 -206791.99649122806) / Math.Sqrt((6306779.07345661 / (20)) + (1718156.318194889 / (2))) is greater than 2.08596344725343 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (2) - 2, .975) and 0.0711106795570928 = (222622.85930107022 - 206791.99649122806) / 222622.85930107022 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 27.191462076916093 (T) = (0 -186262.9517821712) / Math.Sqrt((4302420.009169445 / (20)) + (50455.895731878125 / (2))) is greater than 2.08596344725343 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (2) - 2, .975) and 0.06678930143300904 = (199593.67382756193 - 186262.9517821712) / 199593.67382756193 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 12.82170185002758 (T) = (0 -185852.6607256771) / Math.Sqrt((6664913.895339913 / (20)) + (1764228.2937548775 / (2))) is greater than 2.08596344725343 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (2) - 2, .975) and 0.07067970882450922 = (199987.7356498838 - 185852.6607256771) / 199987.7356498838 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 32.077982243735796 (T) = (0 -218444.31417824072) / Math.Sqrt((4887119.861512551 / (20)) + (163074.4983365718 / (2))) is greater than 2.08596344725343 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (2) - 2, .975) and 0.07734681917204653 = (236756.69115692767 - 218444.31417824072) / 236756.69115692767 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline caa0bd5b30705a2d3f9b67031458c4e468fff4c1
Compare 4dd8a7801fcc2fa888bdde10d2d3ebd1bfe027ac
Diff Diff

Improvements in MicroBenchmarks.Serializers.Binary_FromStream<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BinaryFormatter_ - Duration of single invocation 1.60 ms 1.30 ms 0.82 0.18 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Binary_FromStream&lt;MyEventsListerViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Binary_FromStream<MyEventsListerViewModel>.BinaryFormatter_

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 11.352640732206439 (T) = (0 -1285462.695472435) / Math.Sqrt((10984486399.932228 / (40)) + (682297112.0459615 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.17975704315809699 = (1567173.097616979 - 1285462.695472435) / 1567173.097616979 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline caa0bd5b30705a2d3f9b67031458c4e468fff4c1
Compare 4dd8a7801fcc2fa888bdde10d2d3ebd1bfe027ac
Diff Diff

Improvements in System.Tests.Perf_Environment

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetLogicalDrives - Duration of single invocation 113.81 μs 100.77 μs 0.89 0.09 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Environment*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Environment.GetLogicalDrives

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 14.580657484676884 (T) = (0 -100727.77763755342) / Math.Sqrt((11006730.87239511 / (40)) + (4418.897615030912 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.0708363014363405 = (108406.92312157984 - 100727.77763755342) / 108406.92312157984 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Dec 20, 2022 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline caa0bd5b30705a2d3f9b67031458c4e468fff4c1
Compare 4dd8a7801fcc2fa888bdde10d2d3ebd1bfe027ac
Diff Diff

Improvements in Microsoft.Extensions.Logging.FormattingOverhead

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FourArguments_EnumerableArgument - Duration of single invocation 22.70 μs 18.00 μs 0.79 0.09 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.Logging.FormattingOverhead*'

Payloads

Baseline
Compare

Histogram

Microsoft.Extensions.Logging.FormattingOverhead.FourArguments_EnumerableArgument

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/15/2022 8:20:21 PM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsImprovementStdDev: Marked as improvement because 20.48769406685158 (T) = (0 -17911.239844950076) / Math.Sqrt((1811096.1418834408 / (40)) + (15988.442641829617 / (2))) is greater than 2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (2) - 2, .975) and 0.20886512581068095 = (22639.932114361458 - 17911.239844950076) / 22639.932114361458 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@kotlarmilos kotlarmilos mentioned this issue Apr 4, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-improvement runtime-mono untriaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kotlarmilos