Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regressions in System.Net.Http.Tests.SocketsHttpHandlerPerfTest #66343

Closed
performanceautofiler bot opened this issue Mar 8, 2022 · 11 comments
Closed

Regressions in System.Net.Http.Tests.SocketsHttpHandlerPerfTest #66343

performanceautofiler bot opened this issue Mar 8, 2022 · 11 comments
Assignees
@kouvel
Labels
arch-arm64 arch-x64 area-System.Threading runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Milestone
7.0.0

Comments

@performanceautofiler
Copy link

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9053aa1e66ab7c1af7a14e649fb0bf0b0c501cde
Compare 136b312bc4b471c92eeaf25ded3d0b4dec3afd13
Diff Diff

Regressions in System.Net.Http.Tests.SocketsHttpHandlerPerfTest

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Get - Duration of single invocation 121.45 μs 202.73 μs 1.67 0.54 False
Get_EnumerateHeaders_Unvalidated - Duration of single invocation 134.39 μs 183.62 μs 1.37 0.47 False
Get_EnumerateHeaders_Validated - Duration of single invocation 48.81 μs 54.92 μs 1.13 0.00 True

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Net.Http.Tests.SocketsHttpHandlerPerfTest*'

Payloads

Baseline
Compare

Histogram

System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get(ssl: False, chunkedResponse: True, responseLength: 1)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 202.73157894736843 > 125.80243421052631.
IsChangePoint: Marked as a change because one of 3/4/2022 3:52:45 PM, 3/8/2022 2:08:40 PM falls between 2/27/2022 3:52:50 PM and 3/8/2022 2:08:40 PM.
IsRegressionStdDev: Marked as regression because -18.274347728716307 (T) = (0 -184167.39422084624) / Math.Sqrt((181388575.9425565 / (46)) + (145445285.78867027 / (17))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (17) - 2, .025) and -0.5403656486044143 = (119560.8292016273 - 184167.39422084624) / 119560.8292016273 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get_EnumerateHeaders_Unvalidated(ssl: False, chunkedResponse: True, responseLength: 1)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 183.62 > 127.86513157894737.
IsChangePoint: Marked as a change because one of 3/4/2022 3:52:45 PM, 3/8/2022 2:08:40 PM falls between 2/27/2022 3:52:50 PM and 3/8/2022 2:08:40 PM.
IsRegressionStdDev: Marked as regression because -12.829534994295162 (T) = (0 -179896.05263157896) / Math.Sqrt((234842204.29634947 / (46)) + (252142676.98554683 / (17))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (17) - 2, .025) and -0.46721217324052555 = (122610.79611563987 - 179896.05263157896) / 122610.79611563987 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get_EnumerateHeaders_Validated(ssl: True, chunkedResponse: False, responseLength: 1)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 54.91753515430563 > 52.055332915890716.
IsChangePoint: Marked as a change because one of 3/4/2022 3:52:45 PM, 3/8/2022 2:08:40 PM falls between 2/27/2022 3:52:50 PM and 3/8/2022 2:08:40 PM.
IsRegressionStdDev: Marked as regression because -35.43027244872814 (T) = (0 -55389.85383918084) / Math.Sqrt((435095.83502289344 / (46)) + (291375.36021432554 / (17))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (17) - 2, .025) and -0.11647133480658584 = (49611.532434709945 - 55389.85383918084) / 49611.532434709945 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added CoreClr untriaged New issue has not been triaged by the area owner labels Mar 8, 2022
@kunalspathak kunalspathak changed the title [Perf] Changes at 3/4/2022 7:13:54 PM Regressions in System.Net.Http.Tests.SocketsHttpHandlerPerfTest Mar 8, 2022
@kunalspathak kunalspathak transferred this issue from dotnet/perf-autofiling-issues Mar 8, 2022
@ghost
Copy link

ghost commented Mar 8, 2022

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9053aa1e66ab7c1af7a14e649fb0bf0b0c501cde
Compare 136b312bc4b471c92eeaf25ded3d0b4dec3afd13
Diff Diff

Regressions in System.Net.Http.Tests.SocketsHttpHandlerPerfTest

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Get - Duration of single invocation 121.45 μs 202.73 μs 1.67 0.54 False
Get_EnumerateHeaders_Unvalidated - Duration of single invocation 134.39 μs 183.62 μs 1.37 0.47 False
Get_EnumerateHeaders_Validated - Duration of single invocation 48.81 μs 54.92 μs 1.13 0.00 True

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Net.Http.Tests.SocketsHttpHandlerPerfTest*'

Payloads

Baseline
Compare

Histogram

System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get(ssl: False, chunkedResponse: True, responseLength: 1)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 202.73157894736843 > 125.80243421052631.
IsChangePoint: Marked as a change because one of 3/4/2022 3:52:45 PM, 3/8/2022 2:08:40 PM falls between 2/27/2022 3:52:50 PM and 3/8/2022 2:08:40 PM.
IsRegressionStdDev: Marked as regression because -18.274347728716307 (T) = (0 -184167.39422084624) / Math.Sqrt((181388575.9425565 / (46)) + (145445285.78867027 / (17))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (17) - 2, .025) and -0.5403656486044143 = (119560.8292016273 - 184167.39422084624) / 119560.8292016273 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get_EnumerateHeaders_Unvalidated(ssl: False, chunkedResponse: True, responseLength: 1)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 183.62 > 127.86513157894737.
IsChangePoint: Marked as a change because one of 3/4/2022 3:52:45 PM, 3/8/2022 2:08:40 PM falls between 2/27/2022 3:52:50 PM and 3/8/2022 2:08:40 PM.
IsRegressionStdDev: Marked as regression because -12.829534994295162 (T) = (0 -179896.05263157896) / Math.Sqrt((234842204.29634947 / (46)) + (252142676.98554683 / (17))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (17) - 2, .025) and -0.46721217324052555 = (122610.79611563987 - 179896.05263157896) / 122610.79611563987 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get_EnumerateHeaders_Validated(ssl: True, chunkedResponse: False, responseLength: 1)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 54.91753515430563 > 52.055332915890716.
IsChangePoint: Marked as a change because one of 3/4/2022 3:52:45 PM, 3/8/2022 2:08:40 PM falls between 2/27/2022 3:52:50 PM and 3/8/2022 2:08:40 PM.
IsRegressionStdDev: Marked as regression because -35.43027244872814 (T) = (0 -55389.85383918084) / Math.Sqrt((435095.83502289344 / (46)) + (291375.36021432554 / (17))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (17) - 2, .025) and -0.11647133480658584 = (49611.532434709945 - 55389.85383918084) / 49611.532434709945 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: -
Labels:

area-System.Net.Http, untriaged, refs/heads/main, x64, RunKind=micro, Windows 10.0.18362, Regression, CoreClr
Milestone: -

@kunalspathak
Copy link
Member

Is it introduced in #64834? @kouvel

@kunalspathak kunalspathak added tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark labels Mar 8, 2022
@kouvel
Copy link
Member

kouvel commented Mar 8, 2022

Could be, will take a look

@DrewScoggins
Copy link
Member

We are also seeing a regression on Arm64

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline b463b1630dbf1be5b013208a9fa73e1ecd6c774c
Compare c6ca9dc012bfd1a1fca7b6a2139d20368a258deb
Diff Diff

Regressions in System.Net.Http.Tests.SocketsHttpHandlerPerfTest

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Get - Duration of single invocation 127.62 μs 164.30 μs 1.29 0.30 False
Get - Duration of single invocation 186.08 μs 227.12 μs 1.22 0.21 False
Get_EnumerateHeaders_Validated - Duration of single invocation 123.77 μs 220.28 μs 1.78 0.28 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Net.Http.Tests.SocketsHttpHandlerPerfTest*'

Payloads

Baseline
Compare

Histogram

System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get(ssl: False, chunkedResponse: False, responseLength: 1)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 164.3 > 128.54642269736843.
IsChangePoint: Marked as a change because one of 3/4/2022 10:59:19 PM, 3/6/2022 1:47:20 PM, 3/9/2022 10:55:02 PM falls between 3/1/2022 6:59:52 AM and 3/9/2022 10:55:02 PM.
IsRegressionStdDev: Marked as regression because -3.227335753599241 (T) = (0 -157612.79010434583) / Math.Sqrt((861502638.3159789 / (25)) + (56627685.13621636 / (12))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (12) - 2, .025) and -0.14701005115972163 = (137411.86482628144 - 157612.79010434583) / 137411.86482628144 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get(ssl: False, chunkedResponse: True, responseLength: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 227.1157894736842 > 201.2721052631579.
IsChangePoint: Marked as a change because one of 3/4/2022 11:33:07 AM, 3/9/2022 10:55:02 PM falls between 3/1/2022 6:59:52 AM and 3/9/2022 10:55:02 PM.
IsRegressionStdDev: Marked as regression because -9.08687792208454 (T) = (0 -231749.72828995142) / Math.Sqrt((20314973.700748697 / (19)) + (308525469.8083887 / (18))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (18) - 2, .025) and -0.20093974327871753 = (192973.6521644669 - 231749.72828995142) / 192973.6521644669 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get_EnumerateHeaders_Validated(ssl: False, chunkedResponse: False, responseLength: 1)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 220.28 > 131.25875.
IsChangePoint: Marked as a change because one of 3/4/2022 11:33:07 AM, 3/9/2022 10:55:02 PM falls between 3/1/2022 6:59:52 AM and 3/9/2022 10:55:02 PM.
IsRegressionStdDev: Marked as regression because -6.983531936179113 (T) = (0 -168245.96959446542) / Math.Sqrt((97729291.56483608 / (19)) + (501569613.8648031 / (18))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (18) - 2, .025) and -0.3131557723035305 = (128123.39034182463 - 168245.96959446542) / 128123.39034182463 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@karelz
Copy link
Member

karelz commented Mar 15, 2022

@kouvel did you confirm if it is introduced by that PR? Or should we investigate?

@EgorBo
Copy link
Member

EgorBo commented Mar 15, 2022

Same on Windows-x64 dotnet/perf-autofiling-issues#3987

@kouvel
Copy link
Member

kouvel commented Mar 16, 2022

I'm seeing some regressions from #64834 on my machine. The % diff I see is not as much as above, but I suspect that PR is the main cause.

@kouvel kouvel added area-System.Threading and removed untriaged New issue has not been triaged by the area owner area-System.Net.Http labels Mar 16, 2022
@kouvel kouvel self-assigned this Mar 16, 2022
@kouvel kouvel added this to the 7.0.0 milestone Mar 16, 2022
@ghost
Copy link

ghost commented Mar 16, 2022

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 9053aa1e66ab7c1af7a14e649fb0bf0b0c501cde
Compare 136b312bc4b471c92eeaf25ded3d0b4dec3afd13
Diff Diff

Regressions in System.Net.Http.Tests.SocketsHttpHandlerPerfTest

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Get - Duration of single invocation 121.45 μs 202.73 μs 1.67 0.54 False
Get_EnumerateHeaders_Unvalidated - Duration of single invocation 134.39 μs 183.62 μs 1.37 0.47 False
Get_EnumerateHeaders_Validated - Duration of single invocation 48.81 μs 54.92 μs 1.13 0.00 True

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Net.Http.Tests.SocketsHttpHandlerPerfTest*'

Payloads

Baseline
Compare

Histogram

System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get(ssl: False, chunkedResponse: True, responseLength: 1)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 202.73157894736843 > 125.80243421052631.
IsChangePoint: Marked as a change because one of 3/4/2022 3:52:45 PM, 3/8/2022 2:08:40 PM falls between 2/27/2022 3:52:50 PM and 3/8/2022 2:08:40 PM.
IsRegressionStdDev: Marked as regression because -18.274347728716307 (T) = (0 -184167.39422084624) / Math.Sqrt((181388575.9425565 / (46)) + (145445285.78867027 / (17))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (17) - 2, .025) and -0.5403656486044143 = (119560.8292016273 - 184167.39422084624) / 119560.8292016273 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get_EnumerateHeaders_Unvalidated(ssl: False, chunkedResponse: True, responseLength: 1)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 183.62 > 127.86513157894737.
IsChangePoint: Marked as a change because one of 3/4/2022 3:52:45 PM, 3/8/2022 2:08:40 PM falls between 2/27/2022 3:52:50 PM and 3/8/2022 2:08:40 PM.
IsRegressionStdDev: Marked as regression because -12.829534994295162 (T) = (0 -179896.05263157896) / Math.Sqrt((234842204.29634947 / (46)) + (252142676.98554683 / (17))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (17) - 2, .025) and -0.46721217324052555 = (122610.79611563987 - 179896.05263157896) / 122610.79611563987 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get_EnumerateHeaders_Validated(ssl: True, chunkedResponse: False, responseLength: 1)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 54.91753515430563 > 52.055332915890716.
IsChangePoint: Marked as a change because one of 3/4/2022 3:52:45 PM, 3/8/2022 2:08:40 PM falls between 2/27/2022 3:52:50 PM and 3/8/2022 2:08:40 PM.
IsRegressionStdDev: Marked as regression because -35.43027244872814 (T) = (0 -55389.85383918084) / Math.Sqrt((435095.83502289344 / (46)) + (291375.36021432554 / (17))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (17) - 2, .025) and -0.11647133480658584 = (49611.532434709945 - 55389.85383918084) / 49611.532434709945 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: kouvel
Labels:

arch-arm64, area-System.Threading, tenet-performance, tenet-performance-benchmarks, arch-x64, refs/heads/main, RunKind=micro, Windows 10.0.18362, Regression, CoreClr
Milestone: 7.0.0

@kouvel
Copy link
Member

kouvel commented Mar 17, 2022

There are some known tradeoffs in #64834 where there are few dedicated IO-polling threads like on Unixes, as it introduces an extra thread hop - the polling thread wakes and queues up an IO event, then a worker thread wakes and processes it. That seems to be the difference I'm seeing on my machine, as when DOTNET_SYSTEM_NET_SOCKETS_INLINE_COMPLETIONS=1 (similar implementation on Windows as on Unixes), the poller thread also processes the IO events and the perf is the same as before the change (see the "After + Inline mean" column and the diff column after it in the table below). If the server were to handle multiple connections and queue up incoming requests to the thread pool, that benefit would disappear because that would introduce the same thread hop. If it intends to process requests inline, then the above config option could help. On the other hand, the change has other advantages, for instance less CPU time spent in polling, fewer threads being scheduled / less oversubscription, and in some cases it can be better for latency. Alternative options that may be a more-happy medium are being discussed, currently don't have anything concrete to test with, and those may also come with other tradeoffs. Here are the numbers I'm seeing on my x64 machine:

Method ssl chunked Response response Length Before mean (us) After mean (us) After diff from before After + Inline mean (us) After + Inline diff from before
Get 0 0 1 83.8 99.4 18.7% 83.6 -0.2%
Get 0 0 100000 104.2 126.2 21.1% 106.0 1.7%
Get 0 1 1 83.9 106.4 26.9% 85.0 1.3%
Get 0 1 100000 126.6 148.1 17.0% 127.3 0.5%
Get 1 0 1 102.6 128.9 25.7% 103.9 1.3%
Get 1 0 100000 366.2 350.1 -4.4% 356.3 -2.7%
Get 1 1 1 103.6 133.6 29.0% 104.0 0.5%
Get 1 1 100000 375.1 365.4 -2.6% 360.3 -3.9%
Get_EnumerateHeaders_Unvalidated 0 0 1 83.9 99.6 18.7% 85.1 1.4%
Get_EnumerateHeaders_Unvalidated 0 0 100000 104.1 125.9 21.0% 106.6 2.4%
Get_EnumerateHeaders_Unvalidated 0 1 1 85.3 101.2 18.7% 85.3 0.1%
Get_EnumerateHeaders_Unvalidated 0 1 100000 127.0 147.3 16.0% 126.2 -0.7%
Get_EnumerateHeaders_Unvalidated 1 0 1 101.4 128.5 26.7% 103.9 2.4%
Get_EnumerateHeaders_Unvalidated 1 0 100000 368.0 346.2 -5.9% 365.9 -0.6%
Get_EnumerateHeaders_Unvalidated 1 1 1 103.9 130.8 25.9% 103.8 -0.1%
Get_EnumerateHeaders_Unvalidated 1 1 100000 370.4 378.7 2.3% 359.8 -2.9%
Get_EnumerateHeaders_Validated 0 0 1 84.3 100.0 18.6% 84.5 0.3%
Get_EnumerateHeaders_Validated 0 0 100000 104.1 127.9 22.9% 106.7 2.4%
Get_EnumerateHeaders_Validated 0 1 1 84.7 101.3 19.6% 85.6 1.1%
Get_EnumerateHeaders_Validated 0 1 100000 126.3 146.9 16.3% 127.6 1.0%
Get_EnumerateHeaders_Validated 1 0 1 103.2 129.6 25.5% 102.7 -0.5%
Get_EnumerateHeaders_Validated 1 0 100000 365.2 348.0 -4.7% 359.1 -1.7%
Get_EnumerateHeaders_Validated 1 1 1 103.0 130.0 26.1% 104.8 1.7%
Get_EnumerateHeaders_Validated 1 1 100000 371.4 366.6 -1.3% 361.1 -2.8%

I'll go ahead and close this for now, as based on what is being measured this is an expected difference.

@kouvel kouvel closed this as completed Mar 17, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Apr 16, 2022
@dakersnar
Copy link
Contributor

More regressions here: dotnet/perf-autofiling-issues#3962

@dakersnar
Copy link
Contributor

This regression has shown up on the 7 vs 6 perf report and seems to have persisted on some configs: https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_Windows%2010.0.18362/System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get_EnumerateHeaders_Validated(ssl%3a%20True%2c%20chunkedResponse%3a%20False%2c%20responseLength%3a%201).html.

It seems like these regressions are "by design" based on above explanation.

@jeffhandley jeffhandley added runtime-coreclr specific to the CoreCLR runtime and removed CoreClr labels Dec 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@kouvel kouvel

Labels
arch-arm64 arch-x64 area-System.Threading runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
7.0.0
Development

No branches or pull requests
8 participants
@EgorBo @jeffhandley @karelz @jkotas @DrewScoggins @kunalspathak @kouvel @dakersnar