[Perf] Linux/x64: 2 Regressions on 3/17/2023 1:34:47 AM

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	ef43a7b596545602a8bfe47d7a2b27a8ab5516ae
Compare	673afec8a76cb4437eba318d7132140b7da9b111
Diff	Diff
Configs	AOT:true, CompliationMode:wasm, RunKind:micro

Regressions in System.Collections.CtorFromCollection<Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ImmutableList - Duration of single invocation	117.94 μs	153.16 μs	1.30	0.15	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollection<Int32>*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorFromCollection<Int32>.ImmutableList(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 153.1641187702265 > 127.10563979952798.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 3/16/2023 9:46:36 PM, 3/21/2023 9:37:45 AM falls between 3/12/2023 3:57:11 PM and 3/21/2023 9:37:45 AM.
IsRegressionStdDev: Marked as regression because -20.52790085894532 (T) = (0 -151096.34870768152) / Math.Sqrt((23431541.869968385 / (37)) + (29824185.96488933 / (18))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (18) - 2, .025) and -0.2588136792081109 = (120030.7489530401 - 151096.34870768152) / 120030.7489530401 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	ef43a7b596545602a8bfe47d7a2b27a8ab5516ae
Compare	673afec8a76cb4437eba318d7132140b7da9b111
Diff	Diff
Configs	AOT:true, CompliationMode:wasm, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
SerializeObjectProperty - Duration of single invocation	3.34 μs	3.83 μs	1.15	0.13	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.SerializeObjectProperty(Mode: SourceGen)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.8320803677487594 > 3.5585332488880845.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 3/10/2023 8:06:53 PM, 3/16/2023 9:46:36 PM, 3/21/2023 9:37:45 AM falls between 3/12/2023 3:57:11 PM and 3/21/2023 9:37:45 AM.
IsRegressionStdDev: Marked as regression because -15.259464722174291 (T) = (0 -3819.8877788243326) / Math.Sqrt((13734.430688610266 / (37)) + (10541.093172363564 / (18))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (18) - 2, .025) and -0.14098877422023845 = (3347.8749880206988 - 3819.8877788243326) / 3347.8749880206988 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[radekdoulik]

@kg looks like dotnet/runtime#83490 or dotnet/runtime#83548, it is only small perf change though

[kg]

It looks like ImmutableList is back to exactly where it was in January, which is interesting. I wonder if some sort of BCL change knocked it off a cliff

[kg]

Incidentally I just noticed this is AOT. The second one is JSON so the interpreter might be running code, but ImmutableList should be AOT, which would make that an AOT regression (emscripten bump?)