[mono] iOS HelloWorld size regressions (13 Regressions on 4/9/2024 8:55:31 AM) #100975

performanceautofiler · 2024-04-09T08:55:32Z

Run Information

Name	Value
Architecture	x64
OS	Mac OS X 10.18
Queue	IPhone
Baseline	9b57a265c7efd3732b035bade005561a04767128
Compare	8508806863c840203c5a652667b5f108c3a439f6
Diff	Diff
Configs	CompilationMode:tiered, HybridGlobalization:true, iOSLlvmBuild:false, iOSStripSymbols:false, RunKind:ios_scenarios, RuntimeType:mono

Regressions in SOD - iOS HelloWorld Mono Zip Size nollvm symbols

Benchmark	Baseline	Test	Test/Base	Edge Detector
SOD - iOS HelloWorld Mono Zip Size nollvm symbols - pub/iOSSampleApp.zip ADX - Test Multi Config Graph	5.31 MB	5.51 MB	1.04	True
SOD - iOS HelloWorld Mono Zip Size nollvm symbols - SOD - iOS HelloWorld Mono Zip Size nollvm symbols ADX - Test Multi Config Graph	5.31 MB	5.51 MB	1.04	True
SOD - iOS HelloWorld Mono Zip Size nollvm symbols - pub ADX - Test Multi Config Graph	5.31 MB	5.51 MB	1.04	True
SOD - iOS HelloWorld Mono Zip Size nollvm symbols - Aggregate - .zip ADX - Test Multi Config Graph	5.31 MB	5.51 MB	1.04	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'SOD - iOS HelloWorld Mono Zip Size nollvm symbols*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'SOD - iOS HelloWorld Mono Zip Size nollvm symbols*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'SOD - iOS HelloWorld Mono Zip Size nollvm symbols*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'SOD - iOS HelloWorld Mono Zip Size nollvm symbols*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

SOD - iOS HelloWorld Mono Zip Size nollvm symbols

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Mac OS X 10.18
Queue	IPhone
Baseline	9b57a265c7efd3732b035bade005561a04767128
Compare	8508806863c840203c5a652667b5f108c3a439f6
Diff	Diff
Configs	CompilationMode:tiered, HybridGlobalization:true, iOSLlvmBuild:false, iOSStripSymbols:false, RunKind:ios_scenarios, RuntimeType:mono

Regressions in SOD - iOS HelloWorld Mono .app Size nollvm symbols

Benchmark	Baseline	Test	Test/Base	Edge Detector
app Size nollvm symbols - Aggregate - .dll ADX - Test Multi Config Graph	1.18 MB	1.19 MB	1.01	True
app Size nollvm symbols - pub ADX - Test Multi Config Graph	20.77 MB	22.17 MB	1.07	True
app Size nollvm symbols - SOD - iOS HelloWorld Mono .app Size nollvm symbols ADX - Test Multi Config Graph	20.77 MB	22.17 MB	1.07	True
app Size nollvm symbols - pub/iosHelloWorld/HelloiOS.app/HelloiOS ADX - Test Multi Config Graph	16.57 MB	17.69 MB	1.07	True
app Size nollvm symbols - pub/iosHelloWorld/HelloiOS.app/System.Private.CoreLib.aotdata ADX - Test Multi Config Graph	1.85 MB	1.89 MB	1.02	True
app Size nollvm symbols - Aggregate - No Extension ADX - Test Multi Config Graph	16.57 MB	17.69 MB	1.07	True
app Size nollvm symbols - pub/iosHelloWorld/HelloiOS.app/System.Private.CoreLib.dll ADX - Test Multi Config Graph	1.15 MB	1.16 MB	1.01	True
app Size nollvm symbols - pub/iosHelloWorld/HelloiOS.app/aot-instances.aotdata ADX - Test Multi Config Graph	1.15 MB	1.38 MB	1.20	True
app Size nollvm symbols - Aggregate - .aotdata ADX - Test Multi Config Graph	3.01 MB	3.28 MB	1.09	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'SOD - iOS HelloWorld Mono .app Size nollvm symbols*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'SOD - iOS HelloWorld Mono .app Size nollvm symbols*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'SOD - iOS HelloWorld Mono .app Size nollvm symbols*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'SOD - iOS HelloWorld Mono .app Size nollvm symbols*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

SOD - iOS HelloWorld Mono .app Size nollvm symbols

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

The text was updated successfully, but these errors were encountered:

matouskozak · 2024-04-10T11:17:50Z

Size regressions possibly related to 54c5b47 @tannergooding.

It looks like new generics are causing increase in the AOT generated files. @fanyang-mono

fanyang-mono · 2024-04-19T17:50:10Z

The HelloWorld app size regression is 3.77%. Will see how much impact it is on the Maui iOS app when this change gets populated to the Maui repo.

matouskozak · 2024-05-17T14:52:33Z

I believe MAUI measurements are starting to see this regression but the range of commit is really long so I'm not 100% sure. Here is the issue dotnet/perf-autofiling-issues#33952. It says 5% regression in size but the issue covers also commits from #101488 size regression, so it might be touch lower.

vitek-karas · 2024-07-08T16:43:24Z

We should try to see if there's anything we can do for 9 here. 5% regression is not exactly small...

matouskozak · 2024-07-10T14:26:34Z

Looks like the regression is mostly caused by new generics introduced in #99982 and #101258.

matouskozak · 2024-07-30T15:01:14Z

Closing in favor of #105701, feel free to re-open if you think this issue requires a separate attention.

performanceautofiler bot added arch-x64 runtime-mono specific to the Mono runtime untriaged New issue has not been triaged by the area owner labels Apr 9, 2024

performanceautofiler bot mentioned this issue Apr 9, 2024

[SENTINEL] Autofile run complete at 4/9/2024 8:55:34 AM. 1 issues filed. dotnet/perf-autofiling-issues#32552

Closed

matouskozak mentioned this issue Apr 10, 2024

.NET 9 per-preview performance overview #98209

Open

matouskozak removed the untriaged New issue has not been triaged by the area owner label Apr 10, 2024

matouskozak transferred this issue from dotnet/perf-autofiling-issues Apr 12, 2024

dotnet-issue-labeler bot added the area-Infrastructure-mono label Apr 12, 2024

dotnet-policy-service bot added the untriaged New issue has not been triaged by the area owner label Apr 12, 2024

matouskozak changed the title ~~[Perf] Windows/x64: 13 Regressions on 4/9/2024 8:55:31 AM~~ [mono] iOS HelloWorld size regressions (13 Regressions on 4/9/2024 8:55:31 AM) Apr 19, 2024

matouskozak mentioned this issue May 17, 2024

[Perf] Windows/x64: 1 Regression on 5/7/2024 9:04:47 AM dotnet/perf-autofiling-issues#33952

Closed

matouskozak mentioned this issue Jun 26, 2024

[Perf] Windows/x64: 2 Regressions on 4/11/2024 8:20:38 AM dotnet/perf-autofiling-issues#32624

Closed

vitek-karas added this to the 9.0.0 milestone Jul 8, 2024

vitek-karas removed the untriaged New issue has not been triaged by the area owner label Jul 8, 2024

matouskozak mentioned this issue Jul 11, 2024

[Perf] Linux/arm64: 22 Regressions on 4/8/2024 7:16:22 PM #101127

Open

matouskozak mentioned this issue Jul 30, 2024

[mono] [iOS] Cumulative Mono iOS HelloWorld size regressions from .NET 9 #105701

Open

matouskozak closed this as completed Jul 30, 2024

github-actions bot locked and limited conversation to collaborators Aug 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[mono] iOS HelloWorld size regressions (13 Regressions on 4/9/2024 8:55:31 AM) #100975

[mono] iOS HelloWorld size regressions (13 Regressions on 4/9/2024 8:55:31 AM) #100975

performanceautofiler bot commented Apr 9, 2024 •

edited

Loading

Prerequisites (Build files either built locally or downloaded from payload above)

SOD - iOS HelloWorld Mono Zip Size nollvm symbols

ETL Files

Histogram

JIT Disasms

Docs

Prerequisites (Build files either built locally or downloaded from payload above)

SOD - iOS HelloWorld Mono .app Size nollvm symbols

ETL Files

Histogram

JIT Disasms

Docs

matouskozak commented Apr 10, 2024

fanyang-mono commented Apr 19, 2024

matouskozak commented May 17, 2024 •

edited

Loading

vitek-karas commented Jul 8, 2024

matouskozak commented Jul 10, 2024

matouskozak commented Jul 30, 2024 •

edited

Loading

[mono] iOS HelloWorld size regressions (13 Regressions on 4/9/2024 8:55:31 AM) #100975

[mono] iOS HelloWorld size regressions (13 Regressions on 4/9/2024 8:55:31 AM) #100975

Comments

performanceautofiler bot commented Apr 9, 2024 • edited Loading

Run Information

Regressions in SOD - iOS HelloWorld Mono Zip Size nollvm symbols

Repro

Prerequisites (Build files either built locally or downloaded from payload above)

SOD - iOS HelloWorld Mono Zip Size nollvm symbols

ETL Files

Histogram

JIT Disasms

Docs

Run Information

Regressions in SOD - iOS HelloWorld Mono .app Size nollvm symbols

Repro

Prerequisites (Build files either built locally or downloaded from payload above)

SOD - iOS HelloWorld Mono .app Size nollvm symbols

ETL Files

Histogram

JIT Disasms

Docs

matouskozak commented Apr 10, 2024

fanyang-mono commented Apr 19, 2024

matouskozak commented May 17, 2024 • edited Loading

vitek-karas commented Jul 8, 2024

matouskozak commented Jul 10, 2024

matouskozak commented Jul 30, 2024 • edited Loading

performanceautofiler bot commented Apr 9, 2024 •

edited

Loading

matouskozak commented May 17, 2024 •

edited

Loading

matouskozak commented Jul 30, 2024 •

edited

Loading