Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Regressions from "Remove BBJ_NONE" #95646

Closed
performanceautofiler bot opened this issue Dec 5, 2023 · 9 comments
Closed

[Perf] Regressions from "Remove BBJ_NONE" #95646

performanceautofiler bot opened this issue Dec 5, 2023 · 9 comments
Assignees
@amanasifkhalid
Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) PGO Priority:2 Work that is important, but not critical for the release runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Milestone
9.0.0

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Dec 5, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsFalse<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
14.72 μs 15.81 μs 1.07 0.24 False
18.53 μs 23.86 μs 1.29 0.27 False
18.31 μs 20.77 μs 1.13 0.14 False
16.48 μs 17.81 μs 1.08 0.00 True

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse&lt;Int32&gt;*'

Payloads

Baseline
Compare

System.Collections.ContainsFalse<Int32>.ImmutableHashSet(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Collections.ContainsFalse<Int32>.Queue(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Collections.ContainsFalse<Int32>.ImmutableSortedSet(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Collections.ContainsFalse<Int32>.SortedSet(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Uri

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
32.29 ns 45.63 ns 1.41 0.19 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Uri*'

Payloads

Baseline
Compare

System.Tests.Perf_Uri.EscapeDataString(input: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Int128

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
76.42 ns 93.85 ns 1.23 0.01 True
76.59 ns 93.50 ns 1.22 0.01 True
75.39 ns 94.42 ns 1.25 0.01 True
75.81 ns 94.54 ns 1.25 0.01 True

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int128*'

Payloads

Baseline
Compare

System.Tests.Perf_Int128.TryParse(value: "170141183460469231731687303715884105727")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Int128.TryParseSpan(value: "170141183460469231731687303715884105727")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Int128.ParseSpan(value: "170141183460469231731687303715884105727")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Int128.Parse(value: "170141183460469231731687303715884105727")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Span<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
15.20 ns 16.75 ns 1.10 0.09 False
15.65 ns 21.65 ns 1.38 0.11 False
13.54 ns 15.32 ns 1.13 0.48 False
15.61 ns 23.28 ns 1.49 0.08 True

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Char&gt;*'

Payloads

Baseline
Compare

System.Memory.Span<Char>.Clear(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Memory.Span<Char>.LastIndexOfAnyValues(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Memory.Span<Char>.Reverse(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Memory.Span<Char>.IndexOfAnyThreeValues(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Enum

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
77.66 ns 87.05 ns 1.12 0.13 False
77.70 ns 81.64 ns 1.05 0.10 False
76.41 ns 86.16 ns 1.13 0.12 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Enum*'

Payloads

Baseline
Compare

System.Tests.Perf_Enum.InterpolateIntoSpan_NonFlags(value: 42)

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Enum.InterpolateIntoStringBuilder_Flags(value: 32)

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Enum.InterpolateIntoStringBuilder_NonFlags(value: 42)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Benchstone.MDBenchI.MDAddArray2

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
10.94 ms 12.05 ms 1.10 0.39 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.MDBenchI.MDAddArray2*'

Payloads

Baseline
Compare

Benchstone.MDBenchI.MDAddArray2.Test

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Benchstone.BenchI.CSieve

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
3.52 ms 4.25 ms 1.21 0.00 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchI.CSieve*'

Payloads

Baseline
Compare

Benchstone.BenchI.CSieve.Test

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
36.84 ns 40.61 ns 1.10 0.11 False
82.84 ns 113.58 ns 1.37 0.04 False
32.25 ns 36.42 ns 1.13 0.10 False
32.61 ns 36.20 ns 1.11 0.08 False
2.70 μs 3.57 μs 1.32 0.25 False
37.16 ns 40.65 ns 1.09 0.09 False

graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

System.Tests.Perf_String.ToUpper(s: "This is a much longer piece of text that might benefit more from vectorization.")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_String.IndexerCheckPathLength

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_String.ToUpperInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_String.ToLowerInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_String.Concat_CharEnumerable

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_String.ToLower(s: "This is a much longer piece of text that might benefit more from vectorization.")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IterateFor<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
9.61 μs 10.36 μs 1.08 0.10 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateFor&lt;String&gt;*'

Payloads

Baseline
Compare

System.Collections.IterateFor<String>.ImmutableSortedSet(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added arch-x64 os-linux Linux OS (any supported distro) PGO runtime-coreclr specific to the CoreCLR runtime untriaged New issue has not been triaged by the area owner labels Dec 5, 2023
@performanceautofiler PerformanceAutoFiler
Copy link
Author

performanceautofiler bot commented Dec 5, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsFalse<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
4.89 μs 5.23 μs 1.07 0.03 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse&lt;String&gt;*'

Payloads

Baseline
Compare

System.Collections.ContainsFalse<String>.FrozenSet(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in PerfLabTests.CastingPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
206.14 μs 218.82 μs 1.06 0.02 False
206.25 μs 218.05 μs 1.06 0.03 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.CastingPerf*'

Payloads

Baseline
Compare

PerfLabTests.CastingPerf.ObjFooIsObj2

ETL Files

Histogram

JIT Disasms

PerfLabTests.CastingPerf.IFooFooIsIFoo

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Numerics.Tests.Perf_VectorOf<Int16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
27.51 ns 31.92 ns 1.16 0.03 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;*'

Payloads

Baseline
Compare

System.Numerics.Tests.Perf_VectorOf<Int16>.DivisionOperatorBenchmark

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in SciMark2.kernel

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
543.92 ms 607.67 ms 1.12 0.02 True
576.98 ms 723.01 ms 1.25 0.04 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'SciMark2.kernel*'

Payloads

Baseline
Compare

SciMark2.kernel.benchFFT

ETL Files

Histogram

JIT Disasms

SciMark2.kernel.benchSparseMult

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IterateFor<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
9.25 μs 10.78 μs 1.17 0.33 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateFor&lt;Int32&gt;*'

Payloads

Baseline
Compare

System.Collections.IterateFor<Int32>.ImmutableSortedSet(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Benchstone.BenchI.Array1

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
4.57 ms 5.05 ms 1.10 0.01 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchI.Array1*'

Payloads

Baseline
Compare

Benchstone.BenchI.Array1.Test

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IndexerSet<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
17.50 μs 19.34 μs 1.10 0.00 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IndexerSet&lt;Int32&gt;*'

Payloads

Baseline
Compare

System.Collections.IndexerSet<Int32>.SortedList(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.ReadOnlySpan

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
15.78 ns 19.83 ns 1.26 0.16 True
15.51 ns 19.79 ns 1.28 0.12 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.ReadOnlySpan*'

Payloads

Baseline
Compare

System.Memory.ReadOnlySpan.IndexOfString(input: "だだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだxだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだ", value: "x", comparisonType: Ordinal)

ETL Files

Histogram

JIT Disasms

System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "X", comparisonType: Ordinal)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Perf_Utf8Encoding

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
22.91 μs 27.29 μs 1.19 0.08 False
11.33 μs 13.50 μs 1.19 0.06 False
10.28 μs 12.34 μs 1.20 0.06 False
8.94 μs 10.71 μs 1.20 0.07 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Perf_Utf8Encoding*'

Payloads

Baseline
Compare

System.Text.Perf_Utf8Encoding.GetByteCount(Input: EnglishMostlyAscii)

ETL Files

Histogram

JIT Disasms

System.Text.Perf_Utf8Encoding.GetByteCount(Input: Greek)

ETL Files

Histogram

JIT Disasms

System.Text.Perf_Utf8Encoding.GetByteCount(Input: Chinese)

ETL Files

Histogram

JIT Disasms

System.Text.Perf_Utf8Encoding.GetByteCount(Input: Cyrillic)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.MathBenchmarks.Single

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
7.64 μs 8.61 μs 1.13 0.01 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.MathBenchmarks.Single*'

Payloads

Baseline
Compare

System.MathBenchmarks.Single.ScaleB

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Burgers

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
133.69 ms 175.75 ms 1.31 0.00 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Burgers*'

Payloads

Baseline
Compare

Burgers.Test2

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Encodings.Web.Tests.Perf_Encoders

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
35.59 ns 37.65 ns 1.06 0.09 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Encodings.Web.Tests.Perf_Encoders*'

Payloads

Baseline
Compare

System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: Url,�2020,16)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler performanceautofiler bot mentioned this issue Dec 5, 2023
Closed
@EgorBo EgorBo changed the title [Perf] Linux/x64: 43 Regressions on 11/28/2023 1:58:26 PM [Perf] Regressions from "Remove BBJ_NONE" Dec 5, 2023
@EgorBo EgorBo transferred this issue from dotnet/perf-autofiling-issues Dec 5, 2023
@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Dec 5, 2023
@EgorBo EgorBo added tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Dec 5, 2023
@ghost
Copy link

ghost commented Dec 5, 2023

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsFalse<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
14.72 μs 15.81 μs 1.07 0.24 False
18.53 μs 23.86 μs 1.29 0.27 False
18.31 μs 20.77 μs 1.13 0.14 False
16.48 μs 17.81 μs 1.08 0.00 True

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse&lt;Int32&gt;*'

Payloads

Baseline
Compare

System.Collections.ContainsFalse<Int32>.ImmutableHashSet(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Collections.ContainsFalse<Int32>.Queue(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Collections.ContainsFalse<Int32>.ImmutableSortedSet(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Collections.ContainsFalse<Int32>.SortedSet(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Uri

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
32.29 ns 45.63 ns 1.41 0.19 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Uri*'

Payloads

Baseline
Compare

System.Tests.Perf_Uri.EscapeDataString(input: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Int128

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
76.42 ns 93.85 ns 1.23 0.01 True
76.59 ns 93.50 ns 1.22 0.01 True
75.39 ns 94.42 ns 1.25 0.01 True
75.81 ns 94.54 ns 1.25 0.01 True

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int128*'

Payloads

Baseline
Compare

System.Tests.Perf_Int128.TryParse(value: "170141183460469231731687303715884105727")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Int128.TryParseSpan(value: "170141183460469231731687303715884105727")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Int128.ParseSpan(value: "170141183460469231731687303715884105727")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Int128.Parse(value: "170141183460469231731687303715884105727")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Span<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
15.20 ns 16.75 ns 1.10 0.09 False
15.65 ns 21.65 ns 1.38 0.11 False
13.54 ns 15.32 ns 1.13 0.48 False
15.61 ns 23.28 ns 1.49 0.08 True

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Char&gt;*'

Payloads

Baseline
Compare

System.Memory.Span<Char>.Clear(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Memory.Span<Char>.LastIndexOfAnyValues(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Memory.Span<Char>.Reverse(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Memory.Span<Char>.IndexOfAnyThreeValues(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Enum

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
77.66 ns 87.05 ns 1.12 0.13 False
77.70 ns 81.64 ns 1.05 0.10 False
76.41 ns 86.16 ns 1.13 0.12 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Enum*'

Payloads

Baseline
Compare

System.Tests.Perf_Enum.InterpolateIntoSpan_NonFlags(value: 42)

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Enum.InterpolateIntoStringBuilder_Flags(value: 32)

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Enum.InterpolateIntoStringBuilder_NonFlags(value: 42)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Benchstone.MDBenchI.MDAddArray2

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
10.94 ms 12.05 ms 1.10 0.39 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.MDBenchI.MDAddArray2*'

Payloads

Baseline
Compare

Benchstone.MDBenchI.MDAddArray2.Test

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Benchstone.BenchI.CSieve

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
3.52 ms 4.25 ms 1.21 0.00 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchI.CSieve*'

Payloads

Baseline
Compare

Benchstone.BenchI.CSieve.Test

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
36.84 ns 40.61 ns 1.10 0.11 False
82.84 ns 113.58 ns 1.37 0.04 False
32.25 ns 36.42 ns 1.13 0.10 False
32.61 ns 36.20 ns 1.11 0.08 False
2.70 μs 3.57 μs 1.32 0.25 False
37.16 ns 40.65 ns 1.09 0.09 False

graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

System.Tests.Perf_String.ToUpper(s: "This is a much longer piece of text that might benefit more from vectorization.")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_String.IndexerCheckPathLength

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_String.ToUpperInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_String.ToLowerInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_String.Concat_CharEnumerable

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_String.ToLower(s: "This is a much longer piece of text that might benefit more from vectorization.")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 2158031420214184d22a29029ff24420472fcaa0
Compare e07f80ac344b291b54a3fd5a333906782ab4809e
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IterateFor<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
9.61 μs 10.36 μs 1.08 0.10 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateFor&lt;String&gt;*'

Payloads

Baseline
Compare

System.Collections.IterateFor<String>.ImmutableSortedSet(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: -
Labels:

os-linux, tenet-performance, tenet-performance-benchmarks, arch-x64, area-CodeGen-coreclr, untriaged, runtime-coreclr, PGO
Milestone: -

@EgorBo
Copy link
Member

EgorBo commented Dec 5, 2023

We suspect #94239 cc @amanasifkhalid

@EgorBo
Copy link
Member

EgorBo commented Dec 5, 2023 •
edited by cincuranet
Loading

Also:

x64:

arm64:

@amanasifkhalid
Copy link
Member

Looking at the asmdiffs from that PR, code size is a net improvement, but some collections were a net regression in FullOpts (in particular, benchmarks.run_pgo).

This change is part of a much larger work item that will (hopefully) be a net perf improvement. Is there anything we can/should do to address these transient regressions in the meantime?

@AndyAyersMS
Copy link
Member

I would recommend something like:

  • wait for this Thursday's perf triage results (arm64)
  • wait a bit longer (say a week)
  • build a table collating across all the regression reports: benchmark x arch/os x initial/current results. I have a tool I can share with you to help do this.
  • Investigate the top N entries in this table (note performance fluctuates over time and we want to be able to separate out noise from signal, so the waiting above ensures we have a good sample of data to look at)—especially the largest ones, the most stable ones, and the ones that appear in multiple arch/os results. Write up what you've learned.
  • see if there is any immediate mitigation available. If so work on that or open an issue to remember to work on that. If not, see if there is a reasonable expectation that the rest of the work we plan will eventually fix the problem, and if so, add notes to the overall work item saying we expect it to resolve these specific regressions.

You can look at #84264 or #87194 as examples.

Also look at improvements, since just looking at regressions is going to give you an unbalanced picture of the impact of the change. If the improvements largely outweigh the regressions and the regressions are not things we can easily fix we may just decide to live with things as is.

@AndyAyersMS
Copy link
Member

Also I'll be going through the same process with #95645...

@EgorBo EgorBo mentioned this issue Dec 7, 2023
Closed
@JulieLeeMSFT JulieLeeMSFT added this to the 9.0.0 milestone Dec 8, 2023
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Dec 8, 2023
@amanasifkhalid amanasifkhalid added the Priority:2 Work that is important, but not critical for the release label May 3, 2024
@amanasifkhalid
Copy link
Member

Updated chart:

Notes Recent Score Orig Score Linux arm64 Windows arm64 Linux x64 Windows x64 Windows amd64 Benchmark
1.51 1.55 1.51
1.55		 System.Collections.Sort(BigStruct).List(Size: 512)
1.50 1.31 1.50
1.31		 Burgers.Test2
1.31 1.10 1.31
1.10		 Benchstone.MDBenchI.MDAddArray2.Test
1.27 1.13 1.27
1.13		 System.Tests.Perf_String.ToUpperInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")
1.27 1.16 1.27
1.16		 System.Numerics.Tests.Perf_VectorOf(Int16).DivisionOperatorBenchmark
1.20 1.10 1.20
1.10		 System.Tests.Perf_String.ToUpper(s: "This is a much longer piece of text that might benefit more from vectorization.")
1.19 1.11 1.19
1.11		 System.Tests.Perf_String.ToLowerInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")
1.18 1.12 1.18
1.12		 SciMark2.kernel.benchFFT
1.18 1.13 1.18
1.13		 System.Tests.Perf_Enum.InterpolateIntoStringBuilder_NonFlags(value: 42)
1.17 1.13 1.17
1.13		 System.Memory.Span(Char).Reverse(Size: 512)
1.16 1.09 1.16
1.09		 System.Tests.Perf_String.ToLower(s: "This is a much longer piece of text that might benefit more from vectorization.")
1.16 1.08 1.16
1.08		 System.Collections.IterateFor(String).ImmutableSortedSet(Size: 512)
1.15 1.32 1.15
1.32		 System.Tests.Perf_String.Concat_CharEnumerable
1.12 1.14 1.12
1.14		 System.Collections.IndexerSetReverse(Int32).Span(Size: 512)
1.12 1.10 1.12
1.10		 Benchstone.BenchI.Array1.Test
1.10 1.06 1.10
1.06		 System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: Url,�2020,16)
1.10 1.10 1.10
1.10		 System.Collections.TryGetValueTrue(String, String).ImmutableDictionary(Size: 512)
1.10 1.17 1.10
1.17		 System.Collections.IterateFor(Int32).ImmutableSortedSet(Size: 512)
1.09 1.07 1.09
1.07		 System.Collections.ContainsFalse(String).FrozenSet(Size: 512)
1.09 1.09 1.09
1.09		 System.Tests.Perf_Random.ctor_seeded
1.08 1.07 1.08
1.07		 Benchstone.BenchI.BubbleSort2.Test
1.07 1.06 1.07
1.06		 PerfLabTests.CastingPerf.ObjFooIsObj2
1.07 1.07 1.07
1.07		 System.Collections.ContainsKeyFalse(String, String).ImmutableSortedDictionary(Size: 512)
1.07 1.07 1.07
1.07		 System.Tests.Perf_Double.IsNaN(value: NaN)
1.07 1.07 1.07
1.07		 System.Tests.Perf_Single.IsNaN(value: NaN)
1.07 1.07 1.07
1.07		 System.Tests.Perf_Double.IsNaN(value: 0)
1.07 1.07 1.07
1.07		 System.Tests.Perf_Single.IsNaN(value: 0)
1.07 1.12 1.07
1.12		 System.Tests.Perf_Enum.InterpolateIntoSpan_NonFlags(value: 42)
1.06 1.25 1.06
1.25		 SciMark2.kernel.benchSparseMult
1.06 1.06 1.06
1.06		 System.Text.Perf_Utf8Encoding.GetByteCount(Input: EnglishAllAscii)
1.06 1.06 1.06
1.06		 Benchstone.BenchI.AddArray.Test
1.06 1.06 1.06
1.06		 PerfLabTests.CastingPerf.IFooFooIsIFoo
1.04 1.41 1.04
1.41		 System.Tests.Perf_Uri.EscapeDataString(input: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
1.02 1.23 1.02
1.23		 System.Tests.Perf_Int128.TryParse(value: "170141183460469231731687303715884105727")
1.02 1.07 1.01
1.07		 1.03
1.08		 System.Collections.ContainsFalse(Int32).ImmutableHashSet(Size: 512)
1.02 1.10 1.02
1.10		 System.Collections.IndexerSet(Int32).SortedList(Size: 512)
1.02 1.26 0.78
1.19		 1.33
1.33		 System.Text.Perf_Utf8Encoding.GetByteCount(Input: EnglishMostlyAscii)
1.01 1.14 1.01
1.14		 System.MathBenchmarks.Double.ScaleB
1.01 1.13 1.01
1.13		 System.MathBenchmarks.Single.ScaleB
1.01 1.11 1.09
1.09		 0.93
1.13		 System.Collections.IndexerSet(Int32).Array(Size: 512)
1.01 1.37 1.01
1.37		 System.Tests.Perf_String.IndexerCheckPathLength
1.00 1.11 1.00
1.11		 System.Collections.IndexerSetReverse(Int32).Array(Size: 512)
1.00 1.13 1.06
1.06		 0.94
1.21		 Benchstone.BenchI.CSieve.Test
1.00 1.13 1.00
1.13		 System.Collections.ContainsFalse(Int32).ImmutableSortedSet(Size: 512)
0.99 1.08 0.99
1.08		 System.Collections.ContainsKeyTrue(Int32, Int32).SortedList(Size: 512)
0.99 1.25 0.99
1.25		 System.Tests.Perf_Int128.ParseSpan(value: "170141183460469231731687303715884105727")
0.98 1.40 0.98
1.40		 System.Collections.Sort(BigStruct).Array(Size: 512)
0.98 1.05 0.98
1.05		 System.Tests.Perf_Enum.InterpolateIntoStringBuilder_Flags(value: 32)
0.97 1.08 0.97
1.08		 System.Collections.ContainsFalse(Int32).SortedSet(Size: 512)
0.96 1.22 0.96
1.22		 System.Tests.Perf_Int128.TryParseSpan(value: "170141183460469231731687303715884105727")
0.96 1.25 0.96
1.25		 System.Tests.Perf_Int128.Parse(value: "170141183460469231731687303715884105727")
0.95 1.28 0.95
1.28		 System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
0.94 1.10 0.94
1.10		 System.Collections.ContainsKeyTrue(String, String).ImmutableDictionary(Size: 512)
0.92 1.19 0.75
1.19		 1.13
1.19		 System.Text.Perf_Utf8Encoding.GetByteCount(Input: Greek)
0.88 1.49 0.88
1.49		 System.Memory.Span(Char).IndexOfAnyThreeValues(Size: 512)
0.88 1.26 0.88
1.26		 System.Memory.ReadOnlySpan.IndexOfString(input: "だだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだ
0.88 1.20 0.73
1.20		 1.05
1.21		 System.Text.Perf_Utf8Encoding.GetByteCount(Input: Cyrillic)
0.86 1.33 0.86
1.33		 System.Collections.Sort(BigStruct).LinqQuery(Size: 512)
0.85 1.29 0.85
1.29		 System.Collections.Sort(BigStruct).LinqOrderByExtension(Size: 512)
0.84 1.29 0.84
1.29		 System.Collections.ContainsFalse(Int32).Queue(Size: 512)
0.79 1.20 0.79
1.20		 System.Text.Perf_Utf8Encoding.GetByteCount(Input: Chinese)
0.78 1.38 0.78
1.38		 System.Memory.Span(Char).LastIndexOfAnyValues(Size: 512)
0.76 1.27 0.76
1.27		 System.Collections.ContainsKeyTrue(Int32, Int32).IDictionary(Size: 512)
0.70 1.10 0.70
1.10		 System.Memory.Span(Char).Clear(Size: 512)

@amanasifkhalid
Copy link
Member

I've looked at all of the benchmarks still regressed by >=10% in the above chart, and it's clear that these regressions are from recent churn in block layout (#102763) and/or block compaction (#103972). For example, take a look at the graphs of the top regressions:

image
image
image
image
image
(This one regressed before we turned on the new block layout. I dug into this and found something actionable for .NET 10 here.)

Windows data is split between Windows 10 (purple) and 11 (blue) configs:
System.Collections.Sort<BigStruct>.List(Size: 512)
image

System.Collections.IndexerSetReverse<Int32>.Span(Size: 512) (turquoise is 10, blue is 11)
image

System.Collections.TryGetValueTrue<String, String>.ImmutableDictionary(Size: 512)
image

BBJ_NONE removal doesn't seem relevant to these remaining regressions, so I'm going to retire this issue in favor of #102763 and #103972.

@github-actions github-actions bot locked and limited conversation to collaborators Aug 27, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@amanasifkhalid amanasifkhalid

Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) PGO Priority:2 Work that is important, but not critical for the release runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
9.0.0
Development

No branches or pull requests
4 participants
@EgorBo @AndyAyersMS @amanasifkhalid @JulieLeeMSFT