Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parsing integers has regressed on arm32 #67974

Closed
adamsitnik opened this issue Apr 13, 2022 · 5 comments
Closed

Parsing integers has regressed on arm32 #67974

adamsitnik opened this issue Apr 13, 2022 · 5 comments
Labels
arch-arm32 area-System.Runtime tenet-performance Performance related issue
Milestone
7.0.0

Comments

@adamsitnik
Copy link
Member

Various integer parsing benchmarks have regressed on arm (not on arm64).

Since arm is not covered by the perf lab, we don't have historical data in the Reporting System and all we know is that it happened between preview 2 and 3.

Repro:

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_monthly.py net7.0-preview2 net7.0-preview3 --filter System.Tests.Perf_Int32.Parse --architecture arm

System.Tests.Perf_Int32.Parse(value: "4")

Result Base Diff Ratio Operating System Bit Processor Name
Same 51.09 49.52 1.03 alpine 3.12 Arm64 Unknown processor
Same 44.74 42.88 1.04 ubuntu 18.04 Arm64 Unknown processor
Same 10.27 10.62 0.97 Windows 10 Arm64 Microsoft SQ1 3.0 GHz
Same 10.40 10.74 0.97 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Slower 8.29 11.18 0.74 Windows 11 Arm64 Unknown processor
Same 6.03 5.85 1.03 macOS Monterey 12.3 Arm64 Apple M1 Max
Same 11.31 10.11 1.12 Windows 10 X64 Intel Xeon Platinum 8272CL CPU 2.60GHz
Slower 9.58 13.24 0.72 Windows 10 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Same 8.40 8.12 1.03 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same 6.30 6.66 0.95 Windows 11 X64 AMD Ryzen 9 5900X
Faster 17.81 13.32 1.34 Windows 11 X64 Intel Core i5-4300U CPU 1.90GHz (Haswell)
Faster 13.77 11.00 1.25 Windows 11 X64 Unknown processor
Same 8.62 8.38 1.03 Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Same 9.95 9.48 1.05 Windows 11 X64 Intel Core i9-9900T CPU 2.10GHz
Slower 9.92 15.37 0.65 alpine 3.13 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Faster 26.16 23.34 1.12 centos 7 X64 Intel Xeon CPU E5530 2.40GHz
Same 9.61 9.78 0.98 debian 11 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 10.43 9.73 1.07 opensuse-leap 15.3 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 11.03 11.26 0.98 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Faster 18.41 15.63 1.18 ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Slower 17.64 23.96 0.74 Windows 10 Arm Microsoft SQ1 3.0 GHz
Slower 47.96 58.39 0.82 ubuntu 18.04 Arm ARMv7 Processor rev 3 (v7l)
Same 9.09 8.50 1.07 Windows 10 X86 Intel Xeon CPU E5-1650 v4 3.60GHz
Same 7.44 7.89 0.94 Windows 11 X86 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same 17.27 17.77 0.97 Windows 11 X86 Intel Core i7-10510U CPU 1.80GHz
Same 7.38 7.61 0.97 Windows 7 SP1 X86 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 11.96 12.18 0.98 macOS Monterey 12.2.1 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell)
Same 11.25 10.68 1.05 macOS Monterey 12.3.1 X64 Intel Core i7-4870HQ CPU 2.50GHz (Haswell)

System.Tests.Perf_Int16.TryParse(value: "32767")

Result Base Diff Ratio Operating System Bit Processor Name
Same 70.54 68.96 1.02 alpine 3.12 Arm64 Unknown processor
Same 63.34 62.27 1.02 ubuntu 18.04 Arm64 Unknown processor
Slower 14.76 17.04 0.87 Windows 10 Arm64 Microsoft SQ1 3.0 GHz
Same 14.82 15.24 0.97 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Slower 8.23 15.67 0.53 Windows 11 Arm64 Unknown processor
Same 8.82 8.35 1.06 macOS Monterey 12.3 Arm64 Apple M1 Max
Faster 15.53 13.46 1.15 Windows 10 X64 Intel Xeon Platinum 8272CL CPU 2.60GHz
Same 14.63 15.18 0.96 Windows 10 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Same 11.72 11.29 1.04 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same 8.51 8.61 0.99 Windows 11 X64 AMD Ryzen 9 5900X
Same 19.32 18.25 1.06 Windows 11 X64 Intel Core i5-4300U CPU 1.90GHz (Haswell)
Faster 18.76 14.61 1.28 Windows 11 X64 Unknown processor
Noise 12.59 10.85 - Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Noise 14.69 12.57 - Windows 11 X64 Intel Core i9-9900T CPU 2.10GHz
Same 12.68 12.79 0.99 alpine 3.13 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 35.12 32.00 1.10 centos 7 X64 Intel Xeon CPU E5530 2.40GHz
Same 12.53 12.93 0.97 debian 11 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Noise 12.24 13.93 - opensuse-leap 15.3 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 14.90 15.40 0.97 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Faster 27.86 22.71 1.23 ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Slower 25.73 32.50 0.79 Windows 10 Arm Microsoft SQ1 3.0 GHz
Slower 74.33 84.39 0.88 ubuntu 18.04 Arm ARMv7 Processor rev 3 (v7l)
Same 12.74 11.72 1.09 Windows 10 X86 Intel Xeon CPU E5-1650 v4 3.60GHz
Same 11.19 11.43 0.98 Windows 11 X86 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same 24.99 23.37 1.07 Windows 11 X86 Intel Core i7-10510U CPU 1.80GHz
Same 10.48 9.79 1.07 Windows 7 SP1 X86 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 16.05 15.28 1.05 macOS Monterey 12.2.1 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell)
Same 15.29 14.15 1.08 macOS Monterey 12.3.1 X64 Intel Core i7-4870HQ CPU 2.50GHz (Haswell)

System.Tests.Perf_SByte.TryParse(value: "-128")

Result Base Diff Ratio Operating System Bit Processor Name
Same 61.96 64.00 0.97 alpine 3.12 Arm64 Unknown processor
Same 56.14 55.93 1.00 ubuntu 18.04 Arm64 Unknown processor
Same 12.43 12.67 0.98 Windows 10 Arm64 Microsoft SQ1 3.0 GHz
Same 12.55 12.71 0.99 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Slower 7.65 13.19 0.58 Windows 11 Arm64 Unknown processor
Same 8.19 7.57 1.08 macOS Monterey 12.3 Arm64 Apple M1 Max
Noise 14.27 12.14 - Windows 10 X64 Intel Xeon Platinum 8272CL CPU 2.60GHz
Same 12.65 12.13 1.04 Windows 10 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Same 10.44 9.99 1.05 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same 7.64 7.84 0.98 Windows 11 X64 AMD Ryzen 9 5900X
Same 16.78 15.83 1.06 Windows 11 X64 Intel Core i5-4300U CPU 1.90GHz (Haswell)
Faster 16.03 12.58 1.27 Windows 11 X64 Unknown processor
Noise 11.29 9.85 - Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Noise 12.86 11.07 - Windows 11 X64 Intel Core i9-9900T CPU 2.10GHz
Same 11.62 12.50 0.93 alpine 3.13 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Faster 34.09 28.03 1.22 centos 7 X64 Intel Xeon CPU E5530 2.40GHz
Same 11.89 11.74 1.01 debian 11 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 11.53 11.92 0.97 opensuse-leap 15.3 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 13.67 13.63 1.00 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Faster 24.34 20.20 1.21 ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Slower 22.42 31.26 0.72 Windows 10 Arm Microsoft SQ1 3.0 GHz
Slower 65.23 75.78 0.86 ubuntu 18.04 Arm ARMv7 Processor rev 3 (v7l)
Slower 11.50 14.33 0.80 Windows 10 X86 Intel Xeon CPU E5-1650 v4 3.60GHz
Same 10.60 10.63 1.00 Windows 11 X86 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same 22.08 22.21 0.99 Windows 11 X86 Intel Core i7-10510U CPU 1.80GHz
Same 9.36 9.92 0.94 Windows 7 SP1 X86 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 14.90 14.18 1.05 macOS Monterey 12.2.1 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell)
Faster 17.07 13.16 1.30 macOS Monterey 12.3.1 X64 Intel Core i7-4870HQ CPU 2.50GHz (Haswell)
@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Apr 13, 2022
@ghost
Copy link

ghost commented Apr 13, 2022

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

Issue Details

Various integer parsing benchmarks have regressed on arm (not on arm64).

Since arm is not covered by the perf lab, we don't have historical data in the Reporting System and all we know is that it happened between preview 2 and 3.

Repro:

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_monthly.py net7.0-preview2 net7.0-preview3 --filter System.Tests.Perf_Int32.Parse --architecture arm

System.Tests.Perf_Int32.Parse(value: "4")

Result Base Diff Ratio Operating System Bit Processor Name
Same 51.09 49.52 1.03 alpine 3.12 Arm64 Unknown processor
Same 44.74 42.88 1.04 ubuntu 18.04 Arm64 Unknown processor
Same 10.27 10.62 0.97 Windows 10 Arm64 Microsoft SQ1 3.0 GHz
Same 10.40 10.74 0.97 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Slower 8.29 11.18 0.74 Windows 11 Arm64 Unknown processor
Same 6.03 5.85 1.03 macOS Monterey 12.3 Arm64 Apple M1 Max
Same 11.31 10.11 1.12 Windows 10 X64 Intel Xeon Platinum 8272CL CPU 2.60GHz
Slower 9.58 13.24 0.72 Windows 10 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Same 8.40 8.12 1.03 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same 6.30 6.66 0.95 Windows 11 X64 AMD Ryzen 9 5900X
Faster 17.81 13.32 1.34 Windows 11 X64 Intel Core i5-4300U CPU 1.90GHz (Haswell)
Faster 13.77 11.00 1.25 Windows 11 X64 Unknown processor
Same 8.62 8.38 1.03 Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Same 9.95 9.48 1.05 Windows 11 X64 Intel Core i9-9900T CPU 2.10GHz
Slower 9.92 15.37 0.65 alpine 3.13 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Faster 26.16 23.34 1.12 centos 7 X64 Intel Xeon CPU E5530 2.40GHz
Same 9.61 9.78 0.98 debian 11 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 10.43 9.73 1.07 opensuse-leap 15.3 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 11.03 11.26 0.98 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Faster 18.41 15.63 1.18 ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Slower 17.64 23.96 0.74 Windows 10 Arm Microsoft SQ1 3.0 GHz
Slower 47.96 58.39 0.82 ubuntu 18.04 Arm ARMv7 Processor rev 3 (v7l)
Same 9.09 8.50 1.07 Windows 10 X86 Intel Xeon CPU E5-1650 v4 3.60GHz
Same 7.44 7.89 0.94 Windows 11 X86 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same 17.27 17.77 0.97 Windows 11 X86 Intel Core i7-10510U CPU 1.80GHz
Same 7.38 7.61 0.97 Windows 7 SP1 X86 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 11.96 12.18 0.98 macOS Monterey 12.2.1 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell)
Same 11.25 10.68 1.05 macOS Monterey 12.3.1 X64 Intel Core i7-4870HQ CPU 2.50GHz (Haswell)

System.Tests.Perf_Int16.TryParse(value: "32767")

Result Base Diff Ratio Operating System Bit Processor Name
Same 70.54 68.96 1.02 alpine 3.12 Arm64 Unknown processor
Same 63.34 62.27 1.02 ubuntu 18.04 Arm64 Unknown processor
Slower 14.76 17.04 0.87 Windows 10 Arm64 Microsoft SQ1 3.0 GHz
Same 14.82 15.24 0.97 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Slower 8.23 15.67 0.53 Windows 11 Arm64 Unknown processor
Same 8.82 8.35 1.06 macOS Monterey 12.3 Arm64 Apple M1 Max
Faster 15.53 13.46 1.15 Windows 10 X64 Intel Xeon Platinum 8272CL CPU 2.60GHz
Same 14.63 15.18 0.96 Windows 10 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Same 11.72 11.29 1.04 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same 8.51 8.61 0.99 Windows 11 X64 AMD Ryzen 9 5900X
Same 19.32 18.25 1.06 Windows 11 X64 Intel Core i5-4300U CPU 1.90GHz (Haswell)
Faster 18.76 14.61 1.28 Windows 11 X64 Unknown processor
Noise 12.59 10.85 - Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Noise 14.69 12.57 - Windows 11 X64 Intel Core i9-9900T CPU 2.10GHz
Same 12.68 12.79 0.99 alpine 3.13 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 35.12 32.00 1.10 centos 7 X64 Intel Xeon CPU E5530 2.40GHz
Same 12.53 12.93 0.97 debian 11 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Noise 12.24 13.93 - opensuse-leap 15.3 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 14.90 15.40 0.97 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Faster 27.86 22.71 1.23 ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Slower 25.73 32.50 0.79 Windows 10 Arm Microsoft SQ1 3.0 GHz
Slower 74.33 84.39 0.88 ubuntu 18.04 Arm ARMv7 Processor rev 3 (v7l)
Same 12.74 11.72 1.09 Windows 10 X86 Intel Xeon CPU E5-1650 v4 3.60GHz
Same 11.19 11.43 0.98 Windows 11 X86 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same 24.99 23.37 1.07 Windows 11 X86 Intel Core i7-10510U CPU 1.80GHz
Same 10.48 9.79 1.07 Windows 7 SP1 X86 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 16.05 15.28 1.05 macOS Monterey 12.2.1 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell)
Same 15.29 14.15 1.08 macOS Monterey 12.3.1 X64 Intel Core i7-4870HQ CPU 2.50GHz (Haswell)

System.Tests.Perf_SByte.TryParse(value: "-128")

Result Base Diff Ratio Operating System Bit Processor Name
Same 61.96 64.00 0.97 alpine 3.12 Arm64 Unknown processor
Same 56.14 55.93 1.00 ubuntu 18.04 Arm64 Unknown processor
Same 12.43 12.67 0.98 Windows 10 Arm64 Microsoft SQ1 3.0 GHz
Same 12.55 12.71 0.99 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Slower 7.65 13.19 0.58 Windows 11 Arm64 Unknown processor
Same 8.19 7.57 1.08 macOS Monterey 12.3 Arm64 Apple M1 Max
Noise 14.27 12.14 - Windows 10 X64 Intel Xeon Platinum 8272CL CPU 2.60GHz
Same 12.65 12.13 1.04 Windows 10 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Same 10.44 9.99 1.05 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same 7.64 7.84 0.98 Windows 11 X64 AMD Ryzen 9 5900X
Same 16.78 15.83 1.06 Windows 11 X64 Intel Core i5-4300U CPU 1.90GHz (Haswell)
Faster 16.03 12.58 1.27 Windows 11 X64 Unknown processor
Noise 11.29 9.85 - Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Noise 12.86 11.07 - Windows 11 X64 Intel Core i9-9900T CPU 2.10GHz
Same 11.62 12.50 0.93 alpine 3.13 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Faster 34.09 28.03 1.22 centos 7 X64 Intel Xeon CPU E5530 2.40GHz
Same 11.89 11.74 1.01 debian 11 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 11.53 11.92 0.97 opensuse-leap 15.3 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 13.67 13.63 1.00 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Faster 24.34 20.20 1.21 ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Slower 22.42 31.26 0.72 Windows 10 Arm Microsoft SQ1 3.0 GHz
Slower 65.23 75.78 0.86 ubuntu 18.04 Arm ARMv7 Processor rev 3 (v7l)
Slower 11.50 14.33 0.80 Windows 10 X86 Intel Xeon CPU E5-1650 v4 3.60GHz
Same 10.60 10.63 1.00 Windows 11 X86 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same 22.08 22.21 0.99 Windows 11 X86 Intel Core i7-10510U CPU 1.80GHz
Same 9.36 9.92 0.94 Windows 7 SP1 X86 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 14.90 14.18 1.05 macOS Monterey 12.2.1 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell)
Faster 17.07 13.16 1.30 macOS Monterey 12.3.1 X64 Intel Core i7-4870HQ CPU 2.50GHz (Haswell)
Author: adamsitnik
Assignees: -
Labels:

arch-arm32, area-System.Runtime, tenet-performance
Milestone: -

@adamsitnik adamsitnik mentioned this issue Apr 13, 2022
Closed
18 tasks
@dakersnar
Copy link
Contributor

I'm tempted to call this noise, as these tests are historically noisy on other configs. Unfortunately, we haven't been testing arm32 on perf reports since this one, so it is hard to say if this issue persisted or resolved itself. Given the history of noise, do you think it is safe to call this regression noise? @adamsitnik @tannergooding

image

@dakersnar dakersnar removed the untriaged New issue has not been triaged by the area owner label Jul 15, 2022
@dakersnar dakersnar added this to the 7.0.0 milestone Jul 15, 2022
@dakersnar
Copy link
Contributor

Closing this, I'm assuming this is noise.

@adamsitnik
Copy link
Member Author

@dakersnar could you please run these benchmark for .NET 6 and 7 and see what is the difference?

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 net7.0 --filter System.Tests.Perf_Int32.Parse --architecture arm

@tannergooding
Copy link
Member

This is another example of where even if it was an actual regression, we're looking at ~5ns (somewhere between 5-15 cycles) where other platforms largely stayed the same or had measurable improvements in the inverse direction.

Arm32 is a platform that we only officially support on Linux and which doesn't even have SIMD support/acceleration today. This likely came about as part of the other parsing/formatting improvements and its not something worth spending significant time investigating.

@ghost ghost locked as resolved and limited conversation to collaborators Sep 9, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-arm32 area-System.Runtime tenet-performance Performance related issue
Projects
None yet
Milestone
7.0.0
Development

No branches or pull requests
3 participants
@adamsitnik @tannergooding @dakersnar