Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/arm64: 22 Regressions on 1/5/2023 2:13:22 AM #11482

Closed
performanceautofiler bot opened this issue Jan 10, 2023 · 1 comment
Closed

[Perf] Linux/arm64: 22 Regressions on 1/5/2023 2:13:22 AM #11482

performanceautofiler bot opened this issue Jan 10, 2023 · 1 comment
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Jan 10, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline ee6de921299d1b86a02d9049b9c8dd164cab1401
Compare 8ebfddba8baae0a58957146baf080ee66629baab
Diff Diff

Regressions in System.Text.Json.Document.Tests.Perf_EnumerateObject

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
EnumerateProperties - Duration of single invocation 0.00 ns 73.18 μs ∞ 0.10 False
EnumerateProperties - Duration of single invocation 0.00 ns 73.62 μs ∞ 0.10 False
EnumerateProperties - Duration of single invocation 0.00 ns 78.00 μs ∞ 0.08 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_EnumerateObject*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Document.Tests.Perf_EnumerateObject.EnumerateProperties(TestCase: StringProperties)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 1:41:03 AM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -158.40270219292017 (T) = (0 -72920.34675045066) / Math.Sqrt((0 / (4)) + (847680.6501574708 / (4))) is less than -2.4469118511449675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (4) - 2, .025) and -∞ = (0 - 72920.34675045066) / 0 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Document.Tests.Perf_EnumerateObject.EnumerateProperties(TestCase: NumericProperties)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 1:41:03 AM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -90.80567822022739 (T) = (0 -74864.87219953557) / Math.Sqrt((0 / (4)) + (2718880.7099016216 / (4))) is less than -2.4469118511449675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (4) - 2, .025) and -∞ = (0 - 74864.87219953557) / 0 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Document.Tests.Perf_EnumerateObject.EnumerateProperties(TestCase: ObjectProperties)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 1:41:03 AM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -96.30427287593562 (T) = (0 -78291.66539482315) / Math.Sqrt((0 / (4)) + (2643625.552035349 / (4))) is less than -2.4469118511449675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (4) - 2, .025) and -∞ = (0 - 78291.66539482315) / 0 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline ee6de921299d1b86a02d9049b9c8dd164cab1401
Compare 8ebfddba8baae0a58957146baf080ee66629baab
Diff Diff

Regressions in System.Text.Json.Node.Tests.Perf_ParseThenWrite

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ParseThenWrite - Duration of single invocation 215.15 μs 1.02 ms 4.76 0.04 False
ParseThenWrite - Duration of single invocation 62.99 μs 209.62 μs 3.33 0.08 False
ParseThenWrite - Duration of single invocation 20.90 ms 100.50 ms 4.81 0.06 False
ParseThenWrite - Duration of single invocation 25.83 ms 108.12 ms 4.19 0.06 False
ParseThenWrite - Duration of single invocation 50.49 μs 188.78 μs 3.74 0.04 False
ParseThenWrite - Duration of single invocation 7.49 μs 14.32 μs 1.91 0.04 False
ParseThenWrite - Duration of single invocation 267.31 μs 768.40 μs 2.87 0.13 False
ParseThenWrite - Duration of single invocation 35.19 μs 132.93 μs 3.78 0.04 False
ParseThenWrite - Duration of single invocation 265.14 μs 1.10 ms 4.16 0.06 False
ParseThenWrite - Duration of single invocation 444.40 μs 1.40 ms 3.14 0.04 False
ParseThenWrite - Duration of single invocation 8.01 μs 14.99 μs 1.87 0.05 False
ParseThenWrite - Duration of single invocation 81.22 μs 270.42 μs 3.33 0.04 False
ParseThenWrite - Duration of single invocation 388.10 μs 897.11 μs 2.31 0.04 False
ParseThenWrite - Duration of single invocation 30.13 μs 125.26 μs 4.16 0.08 False
ParseThenWrite - Duration of single invocation 95.99 μs 293.32 μs 3.06 0.08 False
ParseThenWrite - Duration of single invocation 351.20 μs 1.32 ms 3.75 0.09 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Node.Tests.Perf_ParseThenWrite*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: Json4KB)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 1:41:03 AM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -243.97016570777188 (T) = (0 -1015720.5424278919) / Math.Sqrt((442771.50751145324 / (4)) + (42673734.19405583 / (4))) is less than -2.4469118511449675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (4) - 2, .025) and -3.730267203557597 = (214727.94214753373 - 1015720.5424278919) / 214727.94214753373 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: LotsOfStrings)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 1:41:03 AM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -36.02822597326363 (T) = (0 -212260.17910244362) / Math.Sqrt((7992.365405629222 / (4)) + (68755213.03303652 / (4))) is less than -2.4469118511449675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (4) - 2, .025) and -2.375601369913671 = (62880.700604725746 - 212260.17910244362) / 62880.700604725746 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: Json400KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 1:41:03 AM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -232.80543756342968 (T) = (0 -99642143.93809524) / Math.Sqrt((20188112822.73025 / (4)) + (437972860413.554 / (4))) is less than -2.4469118511449675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (4) - 2, .025) and -3.7785528341572236 = (20851949.825865798 - 99642143.93809524) / 20851949.825865798 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: Json400KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 1:41:03 AM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -176.66229178063574 (T) = (0 -107145425.85384616) / Math.Sqrt((1641545370.1597645 / (4)) + (846163516659.649 / (4))) is less than -2.4469118511449675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (4) - 2, .025) and -3.150776000553441 = (25813348.116005294 - 107145425.85384616) / 25813348.116005294 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: LotsOfStrings)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 1:41:03 AM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -53.76578186550379 (T) = (0 -190562.01999741507) / Math.Sqrt((9820.734806054088 / (4)) + (27097542.594052885 / (4))) is less than -2.4469118511449675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (4) - 2, .025) and -2.7662734720245394 = (50596.96843919821 - 190562.01999741507) / 50596.96843919821 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: HelloWorld)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 1:41:03 AM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -83.64040374385564 (T) = (0 -14297.328086690399) / Math.Sqrt((1843.3920858399754 / (4)) + (24736.170898545435 / (4))) is less than -2.4469118511449675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (4) - 2, .025) and -0.911591433512811 = (7479.280266713217 - 14297.328086690399) / 7479.280266713217 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: DeepTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 1:41:03 AM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -166.81017349737402 (T) = (0 -763587.0608207802) / Math.Sqrt((134569.17216415246 / (4)) + (35309006.061291166 / (4))) is less than -2.4469118511449675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (4) - 2, .025) and -1.8594590788610466 = (267038.9887603935 - 763587.0608207802) / 267038.9887603935 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: Json400B)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 1:41:03 AM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -172.97482614839146 (T) = (0 -132392.56601320478) / Math.Sqrt((20829.74845346507 / (4)) + (1240651.3327309557 / (4))) is less than -2.4469118511449675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (4) - 2, .025) and -2.755429214402528 = (35253.64437850757 - 132392.56601320478) / 35253.64437850757 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: Json4KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 1:41:03 AM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -145.29799413193385 (T) = (0 -1094242.6846409682) / Math.Sqrt((576131.7070090704 / (4)) + (129750881.4219083 / (4))) is less than -2.4469118511449675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (4) - 2, .025) and -3.1311584371486147 = (264875.5067830878 - 1094242.6846409682) / 264875.5067830878 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: BroadTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 1:41:03 AM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -154.01268769286187 (T) = (0 -1389956.8924664082) / Math.Sqrt((3326989.575723033 / (4)) + (147176585.5125399 / (4))) is less than -2.4469118511449675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (4) - 2, .025) and -2.1217879835485753 = (445243.8473692973 - 1389956.8924664082) / 445243.8473692973 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: HelloWorld)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 1:41:03 AM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -123.45503444748516 (T) = (0 -14964.117960155481) / Math.Sqrt((4016.303879580971 / (4)) + (8947.500516048469 / (4))) is less than -2.4469118511449675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (4) - 2, .025) and -0.8856226198266153 = (7935.902869860272 - 14964.117960155481) / 7935.902869860272 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: LotsOfNumbers)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 1:41:03 AM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -115.68400648476444 (T) = (0 -269718.5239552132) / Math.Sqrt((2961205.917052807 / (4)) + (7760054.025368792 / (4))) is less than -2.4469118511449675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (4) - 2, .025) and -2.3578606234995583 = (80324.51438502914 - 269718.5239552132) / 80324.51438502914 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: DeepTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 1:41:03 AM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -96.1144665586668 (T) = (0 -892065.7902150296) / Math.Sqrt((51499106.02469194 / (4)) + (63221250.9877299 / (4))) is less than -2.4469118511449675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (4) - 2, .025) and -1.3641130564545085 = (377336.34936768736 - 892065.7902150296) / 377336.34936768736 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: Json400B)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 1:41:03 AM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -47.674645006357984 (T) = (0 -125981.67430593853) / Math.Sqrt((325292.59558378253 / (4)) + (15763137.54556101 / (4))) is less than -2.4469118511449675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (4) - 2, .025) and -3.1483294117278184 = (30369.255139134664 - 125981.67430593853) / 30369.255139134664 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: LotsOfNumbers)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 1:41:03 AM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -134.54365774460325 (T) = (0 -292575.69808298047) / Math.Sqrt((1856963.1725814966 / (4)) + (6706045.600872438 / (4))) is less than -2.4469118511449675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (4) - 2, .025) and -2.0565559487345717 = (95720.7075512909 - 292575.69808298047) / 95720.7075512909 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: BroadTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 1:41:03 AM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -51.62682006418963 (T) = (0 -1269403.2633834572) / Math.Sqrt((1755892.0753134273 / (4)) + (1266661260.3111176 / (4))) is less than -2.4469118511449675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (4) - 2, .025) and -2.6262170863286265 = (350062.67776115634 - 1269403.2633834572) / 350062.67776115634 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline ee6de921299d1b86a02d9049b9c8dd164cab1401
Compare 8ebfddba8baae0a58957146baf080ee66629baab
Diff Diff

Regressions in System.Text.Json.Document.Tests.Perf_EnumerateArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
EnumerateArray - Duration of single invocation 0.00 ns 61.09 μs ∞ 0.12 False
EnumerateArray - Duration of single invocation 0.00 ns 56.04 μs ∞ 0.17 False
EnumerateArray - Duration of single invocation 0.00 ns 57.98 μs ∞ 0.11 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_EnumerateArray*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateArray(TestCase: Json400KB)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 1:41:03 AM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -94.88614354429853 (T) = (0 -60272.1458458642) / Math.Sqrt((0 / (4)) + (1613941.195131798 / (4))) is less than -2.4469118511449675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (4) - 2, .025) and -∞ = (0 - 60272.1458458642) / 0 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateArray(TestCase: ArrayOfNumbers)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 1:41:03 AM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -44.78478285080438 (T) = (0 -57239.07716338092) / Math.Sqrt((0 / (4)) + (6534077.664684057 / (4))) is less than -2.4469118511449675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (4) - 2, .025) and -∞ = (0 - 57239.07716338092) / 0 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateArray(TestCase: ArrayOfStrings)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 1:41:03 AM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -104.2400191798823 (T) = (0 -56929.52416576345) / Math.Sqrt((0 / (4)) + (1193070.572529292 / (4))) is less than -2.4469118511449675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (4) - 2, .025) and -∞ = (0 - 56929.52416576345) / 0 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added ampere arm64 PGO Applied if there were any profile guided optimization updates in the observed interval. untriaged labels Jan 10, 2023
@kotlarmilos
Copy link
Member

Duplicate of #11449

@kotlarmilos kotlarmilos marked this as a duplicate of #11449 Jan 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kotlarmilos