Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BenchmarksGame.BinaryTrees benchmarks have regressed #67958

Closed
adamsitnik opened this issue Apr 13, 2022 · 6 comments
Closed

BenchmarksGame.BinaryTrees benchmarks have regressed #67958

adamsitnik opened this issue Apr 13, 2022 · 6 comments
Assignees
@mrsharm
Labels
area-GC-coreclr tenet-performance Performance related issue
Milestone
8.0.0

Comments

@adamsitnik
Copy link
Member

BenchmarksGame.BinaryTrees_2.RunBench, BenchmarksGame.BinaryTrees_5.RunBench and BenchmarksGame.BinaryTrees_6.RunBench have regressed across multiple configurations.

Reporting System links:

image

image

For BinaryTrees_5 looks like there was more than one regression:

image

Other two benchmarks were most likely regressed by a single change.

It could have been #65675 (cc @AndyAyersMS) but also #64688 (cc @mangod9)

Repro:

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_monthly.py net7.0-preview2 net7.0-preview3 --filter BenchmarksGame.BinaryTrees*

BenchmarksGame.BinaryTrees_5.RunBench

Result Base Diff Ratio Operating System Bit
Slower 393541960.00 516028482.00 0.76 alpine 3.12 Arm64
Same 219965100.00 206004000.00 1.07 debian 11 Arm64
Slower 381863860.00 609215398.50 0.63 ubuntu 18.04 Arm64
Slower 172751300.00 204697050.00 0.84 ubuntu 20.04 Arm64
Slower 155414700.00 222475250.00 0.70 Windows 10 Arm64
Slower 155923400.00 204938200.00 0.76 Windows 11 Arm64
Faster 156545937.50 113752271.00 1.38 macOS Monterey 12.3 Arm64
Slower 106274800.00 144639075.00 0.73 Windows 10 X64
Slower 105588975.00 135448650.00 0.78 Windows 11 X64
Slower 48734500.00 55338512.50 0.88 Windows 11 X64
Same 198271700.00 199862600.00 0.99 Windows 11 X64
Slower 202873800.00 380059600.00 0.53 Windows 11 X64
Slower 116493850.00 158733150.00 0.73 Windows 11 X64
Slower 114900700.00 155972500.00 0.74 Windows 11 X64
Slower 132087500.00 187532200.00 0.70 alpine 3.13 X64
Slower 244331335.00 316001823.00 0.77 centos 7 X64
Slower 125837625.00 161531250.00 0.78 debian 11 X64
Slower 132354850.00 163942000.00 0.81 opensuse-leap 15.3 X64
Same 152814471.00 164402935.50 0.93 pop 20.04 X64
Slower 193255715.00 236016529.00 0.82 ubuntu 18.04 X64
Slower 266200033.50 359426625.00 0.74 ubuntu 18.04 X64
Same 125708100.00 125210550.00 1.00 Windows 10 Arm
Same 467789485.00 451069876.00 1.04 ubuntu 18.04 Arm
Same 86750575.00 86725725.00 1.00 Windows 10 X86
Same 83515875.00 82701300.00 1.01 Windows 11 X86
Slower 230724300.00 509935600.00 0.45 Windows 11 X86
Same 60702850.00 62364225.00 0.97 Windows 7 SP1 X86
Same 264454245.00 266784088.00 0.99 macOS Monterey 12.2.1 X64
Same 233291064.00 241275843.50 0.97 macOS Monterey 12.3.1 X64

BenchmarksGame.BinaryTrees_6.RunBench

Result Base Diff Ratio Operating System Bit
Slower 1767771555.50 2085189207.00 0.85 alpine 3.12 Arm64
Same 882082950.00 799602450.00 1.10 debian 11 Arm64
Slower 1710713842.00 2015879750.50 0.85 ubuntu 18.04 Arm64
Same 725956000.00 856108600.00 0.85 ubuntu 20.04 Arm64
Slower 722680250.00 945867900.00 0.76 Windows 10 Arm64
Slower 738982100.00 887822650.00 0.83 Windows 11 Arm64
Same 431022584.00 462372833.00 0.93 macOS Monterey 12.3 Arm64
Slower 535746100.00 637023450.00 0.84 Windows 10 X64
Same 559887900.00 604014350.00 0.93 Windows 11 X64
Slower 292341500.00 333542050.00 0.88 Windows 11 X64
Same 976115700.00 1103100400.00 0.88 Windows 11 X64
Slower 933505700.00 1517357100.00 0.62 Windows 11 X64
Slower 490084100.00 636806300.00 0.77 Windows 11 X64
Same 610569000.00 680544500.00 0.90 Windows 11 X64
Slower 539701400.00 683311300.00 0.79 alpine 3.13 X64
Slower 1063519943.00 1250318351.50 0.85 centos 7 X64
Slower 525434650.00 614036200.00 0.86 debian 11 X64
Slower 540644200.00 615633200.00 0.88 opensuse-leap 15.3 X64
Slower 722297588.00 856043703.00 0.84 pop 20.04 X64
Slower 746559238.00 868726827.00 0.86 ubuntu 18.04 X64
Slower 928109367.50 1147816790.50 0.81 ubuntu 18.04 X64
Same 696386200.00 688536650.00 1.01 Windows 10 Arm
Same 2805719008.00 2835854384.00 0.99 ubuntu 18.04 Arm
Same 514031700.00 495507900.00 1.04 Windows 10 X86
Same 473811700.00 474432850.00 1.00 Windows 11 X86
Slower 1229425850.00 1998982550.00 0.62 Windows 11 X86
Same 499968300.00 507812700.00 0.98 Windows 7 SP1 X86
Same 1495033756.00 1573272813.50 0.95 macOS Monterey 12.2.1 X64
Same 1327254503.00 1407238203.00 0.94 macOS Monterey 12.3.1 X64

BenchmarksGame.BinaryTrees_2.RunBench

Result Base Diff Ratio Operating System Bit
Same 518957423.50 548738312.00 0.95 alpine 3.12 Arm64
Faster 235095900.00 196742650.00 1.19 debian 11 Arm64
Same 484596457.00 504593832.00 0.96 ubuntu 18.04 Arm64
Faster 294381500.00 199778950.00 1.47 ubuntu 20.04 Arm64
Same 190647950.00 199972600.00 0.95 Windows 10 Arm64
Same 188582700.00 193668300.00 0.97 Windows 11 Arm64
Same 102197458.50 95341604.00 1.07 macOS Monterey 12.3 Arm64
Same 110957300.00 116475600.00 0.95 Windows 10 X64
Same 92762100.00 94503675.00 0.98 Windows 11 X64
Same 75655850.00 72425662.50 1.04 Windows 11 X64
Same 192509500.00 198119200.00 0.97 Windows 11 X64
Slower 158180950.00 277990600.00 0.57 Windows 11 X64
Same 98589600.00 95628350.00 1.03 Windows 11 X64
Same 112493550.00 117155000.00 0.96 Windows 11 X64
Same 135793725.00 128776550.00 1.05 alpine 3.13 X64
Same 288341906.00 267079383.50 1.08 centos 7 X64
Same 124722200.00 130884225.00 0.95 debian 11 X64
Same 136604000.00 133805525.00 1.02 opensuse-leap 15.3 X64
Same 182588100.00 169489884.00 1.08 pop 20.04 X64
Same 127584650.00 123601525.50 1.03 ubuntu 18.04 X64
Faster 231406396.00 189891688.50 1.22 ubuntu 18.04 X64
Same 194641600.00 188638900.00 1.03 Windows 10 Arm
Same 782469453.00 726321563.00 1.08 ubuntu 18.04 Arm
Same 150418050.00 159450200.00 0.94 Windows 10 X86
Same 129401075.00 129430825.00 1.00 Windows 11 X86
Slower 187704600.00 505429500.00 0.37 Windows 11 X86
Same 92401000.00 91051450.00 1.01 Windows 7 SP1 X86
Same 234774368.00 232889694.00 1.01 macOS Monterey 12.2.1 X64
Same 232508956.00 232856854.00 1.00 macOS Monterey 12.3.1 X64
@adamsitnik adamsitnik added the tenet-performance Performance related issue label Apr 13, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Apr 13, 2022
@adamsitnik adamsitnik mentioned this issue Apr 13, 2022
Closed
@adamsitnik
Copy link
Member Author

ByteMark.BenchBitOps has also regressed (and partially recovered) and the diff points to GC regions so I am going to move it to GC area.

image

@ghost
Copy link

ghost commented Apr 13, 2022

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

BenchmarksGame.BinaryTrees_2.RunBench, BenchmarksGame.BinaryTrees_5.RunBench and BenchmarksGame.BinaryTrees_6.RunBench have regressed across multiple configurations.

Reporting System links:

image

image

For BinaryTrees_5 looks like there was more than one regression:

image

Other two benchmarks were most likely regressed by a single change.

It could have been #65675 (cc @AndyAyersMS) but also #64688 (cc @mangod9)

Repro:

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_monthly.py net7.0-preview2 net7.0-preview3 --filter BenchmarksGame.BinaryTrees*

BenchmarksGame.BinaryTrees_5.RunBench

Result Base Diff Ratio Operating System Bit
Slower 393541960.00 516028482.00 0.76 alpine 3.12 Arm64
Same 219965100.00 206004000.00 1.07 debian 11 Arm64
Slower 381863860.00 609215398.50 0.63 ubuntu 18.04 Arm64
Slower 172751300.00 204697050.00 0.84 ubuntu 20.04 Arm64
Slower 155414700.00 222475250.00 0.70 Windows 10 Arm64
Slower 155923400.00 204938200.00 0.76 Windows 11 Arm64
Faster 156545937.50 113752271.00 1.38 macOS Monterey 12.3 Arm64
Slower 106274800.00 144639075.00 0.73 Windows 10 X64
Slower 105588975.00 135448650.00 0.78 Windows 11 X64
Slower 48734500.00 55338512.50 0.88 Windows 11 X64
Same 198271700.00 199862600.00 0.99 Windows 11 X64
Slower 202873800.00 380059600.00 0.53 Windows 11 X64
Slower 116493850.00 158733150.00 0.73 Windows 11 X64
Slower 114900700.00 155972500.00 0.74 Windows 11 X64
Slower 132087500.00 187532200.00 0.70 alpine 3.13 X64
Slower 244331335.00 316001823.00 0.77 centos 7 X64
Slower 125837625.00 161531250.00 0.78 debian 11 X64
Slower 132354850.00 163942000.00 0.81 opensuse-leap 15.3 X64
Same 152814471.00 164402935.50 0.93 pop 20.04 X64
Slower 193255715.00 236016529.00 0.82 ubuntu 18.04 X64
Slower 266200033.50 359426625.00 0.74 ubuntu 18.04 X64
Same 125708100.00 125210550.00 1.00 Windows 10 Arm
Same 467789485.00 451069876.00 1.04 ubuntu 18.04 Arm
Same 86750575.00 86725725.00 1.00 Windows 10 X86
Same 83515875.00 82701300.00 1.01 Windows 11 X86
Slower 230724300.00 509935600.00 0.45 Windows 11 X86
Same 60702850.00 62364225.00 0.97 Windows 7 SP1 X86
Same 264454245.00 266784088.00 0.99 macOS Monterey 12.2.1 X64
Same 233291064.00 241275843.50 0.97 macOS Monterey 12.3.1 X64

BenchmarksGame.BinaryTrees_6.RunBench

Result Base Diff Ratio Operating System Bit
Slower 1767771555.50 2085189207.00 0.85 alpine 3.12 Arm64
Same 882082950.00 799602450.00 1.10 debian 11 Arm64
Slower 1710713842.00 2015879750.50 0.85 ubuntu 18.04 Arm64
Same 725956000.00 856108600.00 0.85 ubuntu 20.04 Arm64
Slower 722680250.00 945867900.00 0.76 Windows 10 Arm64
Slower 738982100.00 887822650.00 0.83 Windows 11 Arm64
Same 431022584.00 462372833.00 0.93 macOS Monterey 12.3 Arm64
Slower 535746100.00 637023450.00 0.84 Windows 10 X64
Same 559887900.00 604014350.00 0.93 Windows 11 X64
Slower 292341500.00 333542050.00 0.88 Windows 11 X64
Same 976115700.00 1103100400.00 0.88 Windows 11 X64
Slower 933505700.00 1517357100.00 0.62 Windows 11 X64
Slower 490084100.00 636806300.00 0.77 Windows 11 X64
Same 610569000.00 680544500.00 0.90 Windows 11 X64
Slower 539701400.00 683311300.00 0.79 alpine 3.13 X64
Slower 1063519943.00 1250318351.50 0.85 centos 7 X64
Slower 525434650.00 614036200.00 0.86 debian 11 X64
Slower 540644200.00 615633200.00 0.88 opensuse-leap 15.3 X64
Slower 722297588.00 856043703.00 0.84 pop 20.04 X64
Slower 746559238.00 868726827.00 0.86 ubuntu 18.04 X64
Slower 928109367.50 1147816790.50 0.81 ubuntu 18.04 X64
Same 696386200.00 688536650.00 1.01 Windows 10 Arm
Same 2805719008.00 2835854384.00 0.99 ubuntu 18.04 Arm
Same 514031700.00 495507900.00 1.04 Windows 10 X86
Same 473811700.00 474432850.00 1.00 Windows 11 X86
Slower 1229425850.00 1998982550.00 0.62 Windows 11 X86
Same 499968300.00 507812700.00 0.98 Windows 7 SP1 X86
Same 1495033756.00 1573272813.50 0.95 macOS Monterey 12.2.1 X64
Same 1327254503.00 1407238203.00 0.94 macOS Monterey 12.3.1 X64

BenchmarksGame.BinaryTrees_2.RunBench

Result Base Diff Ratio Operating System Bit
Same 518957423.50 548738312.00 0.95 alpine 3.12 Arm64
Faster 235095900.00 196742650.00 1.19 debian 11 Arm64
Same 484596457.00 504593832.00 0.96 ubuntu 18.04 Arm64
Faster 294381500.00 199778950.00 1.47 ubuntu 20.04 Arm64
Same 190647950.00 199972600.00 0.95 Windows 10 Arm64
Same 188582700.00 193668300.00 0.97 Windows 11 Arm64
Same 102197458.50 95341604.00 1.07 macOS Monterey 12.3 Arm64
Same 110957300.00 116475600.00 0.95 Windows 10 X64
Same 92762100.00 94503675.00 0.98 Windows 11 X64
Same 75655850.00 72425662.50 1.04 Windows 11 X64
Same 192509500.00 198119200.00 0.97 Windows 11 X64
Slower 158180950.00 277990600.00 0.57 Windows 11 X64
Same 98589600.00 95628350.00 1.03 Windows 11 X64
Same 112493550.00 117155000.00 0.96 Windows 11 X64
Same 135793725.00 128776550.00 1.05 alpine 3.13 X64
Same 288341906.00 267079383.50 1.08 centos 7 X64
Same 124722200.00 130884225.00 0.95 debian 11 X64
Same 136604000.00 133805525.00 1.02 opensuse-leap 15.3 X64
Same 182588100.00 169489884.00 1.08 pop 20.04 X64
Same 127584650.00 123601525.50 1.03 ubuntu 18.04 X64
Faster 231406396.00 189891688.50 1.22 ubuntu 18.04 X64
Same 194641600.00 188638900.00 1.03 Windows 10 Arm
Same 782469453.00 726321563.00 1.08 ubuntu 18.04 Arm
Same 150418050.00 159450200.00 0.94 Windows 10 X86
Same 129401075.00 129430825.00 1.00 Windows 11 X86
Slower 187704600.00 505429500.00 0.37 Windows 11 X86
Same 92401000.00 91051450.00 1.01 Windows 7 SP1 X86
Same 234774368.00 232889694.00 1.01 macOS Monterey 12.2.1 X64
Same 232508956.00 232856854.00 1.00 macOS Monterey 12.3.1 X64
Author: adamsitnik
Assignees: -
Labels:

tenet-performance, area-GC-coreclr, untriaged
Milestone: -

@adamsitnik
Copy link
Member Author

Another example of benchmark that has regressed is System.Tests.Perf_GC<Byte>.NewOperator_Array(length: 1000):

image

@adamsitnik adamsitnik mentioned this issue Apr 13, 2022
Closed
18 tasks
@mangod9 mangod9 removed the untriaged New issue has not been triaged by the area owner label Jul 7, 2022
@mangod9 mangod9 added this to the 7.0.0 milestone Jul 7, 2022
@mrsharm mrsharm mentioned this issue Aug 8, 2022
Closed
@mrsharm mrsharm modified the milestones: 7.0.0, 8.0.0 Aug 8, 2022
@mrsharm
Copy link
Member

mrsharm commented Aug 8, 2022

Attribute these GC related regressions to the decommit behavior for regions that's being tracked; #73592 & updated the milestone.

@mrsharm
Copy link
Member

mrsharm commented Aug 5, 2023

Closing in lieu of #73592 with the latest Net 8.0 results.

@mrsharm mrsharm closed this as completed Aug 5, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Sep 4, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@mrsharm mrsharm

Labels
area-GC-coreclr tenet-performance Performance related issue
Projects
None yet
Milestone
8.0.0
Development

No branches or pull requests
3 participants
@adamsitnik @mangod9 @mrsharm