Size regression in Minimal API NativeAOT app

[eerhardt]

Description

Between builds 8.0.0-preview.2.23123.4+ff7c19f and 8.0.0-preview.2.23123.5+5ba867f there was a ~140KB size regression on linux-x64 Minimal API NativeAOT app.

Configuration

• Which version of .NET is the code running on? .NET 8
• What OS version, and what distro if applicable? Linux
• What is the architecture (x64, x86, ARM, ARM64)? x64

Regression?

Yes. From a previous build.

Data

Here are the crank commands to repro:

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicminimalapipublishaot --profile intel-load2-app --profile amd-lin2-load --profile amd-lin2-db --application.packageReferences Microsoft.Dotnet.ILCompiler=8.0.0-preview.2.23123.4 --application.framework net8.0 --application.options.collectCounters true --application.aspNetCoreVersion 8.0.0-preview.2.23123.5 --application.runtimeVersion 8.0.0-preview.2.23123.4 --application.sdkVersion 8.0.100-preview.2.23124.1

and

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicminimalapipublishaot --profile intel-load2-app --profile amd-lin2-load --profile amd-lin2-db --application.packageReferences Microsoft.Dotnet.ILCompiler=8.0.0-preview.2.23123.5 --application.framework net8.0 --application.options.collectCounters true --application.aspNetCoreVersion 8.0.0-preview.2.23123.5 --application.runtimeVersion 8.0.0-preview.2.23123.5 --application.sdkVersion 8.0.100-preview.2.23124.1

Analysis

By diffing the mstat files between these two builds, it appears the majority of the size regression comes from the System namespace.

And drilling into the System namespace, I see a bunch of SZGenericArrayEnumerator`1 types that weren't there before, but are now.

And inspecting the diff between the two builds: ff7c19f...5ba867f, this leads me to believe the regression was caused by #82499. (cc @stephentoub)

I've zipped the before and after mstat files, along with a text dump in BasicMinimalApi.zip, if anyone wants to take a look.

[ghost]

Tagging subscribers to this area: @dotnet/area-system-collections
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Between builds 8.0.0-preview.2.23123.4+ff7c19f and 8.0.0-preview.2.23123.5+5ba867f there was a ~140KB size regression on linux-x64 Minimal API NativeAOT app.

Configuration

• Which version of .NET is the code running on? .NET 8
• What OS version, and what distro if applicable? Linux
• What is the architecture (x64, x86, ARM, ARM64)? x64

Regression?

Yes. From a previous build.

Data

Here are the crank commands to repro:

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicminimalapipublishaot --profile intel-load2-app --profile amd-lin2-load --profile amd-lin2-db --application.packageReferences Microsoft.Dotnet.ILCompiler=8.0.0-preview.2.23123.4 --application.framework net8.0 --application.options.collectCounters true --application.aspNetCoreVersion 8.0.0-preview.2.23123.5 --application.runtimeVersion 8.0.0-preview.2.23123.4 --application.sdkVersion 8.0.100-preview.2.23124.1

and

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicminimalapipublishaot --profile intel-load2-app --profile amd-lin2-load --profile amd-lin2-db --application.packageReferences Microsoft.Dotnet.ILCompiler=8.0.0-preview.2.23123.5 --application.framework net8.0 --application.options.collectCounters true --application.aspNetCoreVersion 8.0.0-preview.2.23123.5 --application.runtimeVersion 8.0.0-preview.2.23123.5 --application.sdkVersion 8.0.100-preview.2.23124.1

Analysis

By diffing the mstat files between these two builds, it appears the majority of the size regression comes from the System namespace.

And drilling into the System namespace, I see a bunch of SZGenericArrayEnumerator`1 types that weren't there before, but are now.

And inspecting the diff between the two builds: ff7c19f...5ba867f, this leads me to believe the regression was caused by #82499. (cc @stephentoub)

I've zipped the before and after mstat files, along with a text dump in BasicMinimalApi.zip, if anyone wants to take a look.

Author:	eerhardt
Assignees:	-
Labels:	area-System.Collections, tenet-performance
Milestone:	-

[stephentoub]

What would you suggest be done here, @eerhardt? The PR makes it so that enumerating an empty collection (for our core collection types) returns a singleton enumerator, saving the associated allocation costs. Is 140KB in the minimal API app worth the tradeoff?

[eerhardt]

What would you suggest be done here, @eerhardt?

I'm not sure. I mostly wanted to have a conversation on if there is anything we could do here.

Is there something smarter we can do either in IL, or in the ILC, to share/reuse code here? cc @MichalStrehovsky

A lot of the entries appear to be System.SZGenericArrayEnumerator`1<System.Collections.Generic.KeyValuePair`2. Maybe that one isn't common enough to need to optimize enumerating an empty collection?

Is 140KB in the minimal API app worth the tradeoff?

It represents about 1% of the app right now, and hopefully will be more like 1.5% of the app by the time .NET 8 ships. So, not huge.

[MichalStrehovsky]

The SZArray types are unshareable. NativeAOT uses different array enumerators that are size optimized:

runtime/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Array.NativeAot.cs

Line 1301 in eccbe90

private sealed class ArrayEnumerator : ArrayEnumeratorBase, IEnumerator<T>

The SZArray enumerator type implements array enumerators in CoreCLR only.

It would be good not to use array enumerators for this in general (not just "don't use CoreCLR's array enumerator in NativeAOT"). Array enumerators are covariant and have a bunch of magic pixie dust that makes them harder to optimize.

[MichalStrehovsky]

Array enumerators are covariant

I meant "array covariant" - so IEnumerator<int> is castable to IEnumerator<uint>. I guess we would want to measure, maybe using array enumerators would be still fine because we get sharing, but it should at minimum be the real array enumerators in the CoreLib.

[stephentoub]

NativeAOT uses different array enumerators that are size optimized

Does that suggest that if we changed the call sites I added in corelib from:

SZGenericArrayEnumerator<T>.Empty

to

((IEnumerable<T>)EmptyArray<T>.Value).GetEnumerator()

or something similar, it would do the right thing? Or does that open its own can of worms due to the possibly additional reference to EmptyArray<T>?

[jkotas]

The problem is that each T[] instantiation is a ton of footprint. Piggy backing on T[] empty enumerator is good for footprint only if the code uses T[] for other reasons. If the code does not use T[] for other reasons, it would be best to use dedicated empty enumerator that does not reference T[].

The SZArray types are unshareable. NativeAOT uses different array enumerators that are size optimized:

There is no good reason for this duplication. We should be able to unify these implementations.

[MichalStrehovsky]

There is no good reason for this duplication. We should be able to unify these implementations.

That might be the best solution.

[jkotas]

WIP: #82751

[stephentoub]

If the code does not use T[] for other reasons, it would be best to use dedicated empty enumerator that does not reference T[].

Is this what you had in mind?
main...stephentoub:runtime:emptyenumerator

I wouldn't expect that to help with the diff Eric showed, though; all of those extra instantiations are from a new use of the array enumerator type, rather than from T[], at least if I'm understanding the diff correctly.

[jkotas]

Is this what you had in mind?

Yes, something like that. I would also move methods that do not depend on generics to non-generic base type, similar to how it is done for regular array iterator after my PR.

I have done some ad-hoc analysis to find array types instantiated just for the empty enumerator in BasicMinimalApi. Here is the list:

<class [Microsoft.AspNetCore.Http.Extensions]Microsoft.AspNetCore.Http.ParameterBindingMethodCache/ParameterLookupKey>
<class [Microsoft.AspNetCore.Routing.Abstractions]Microsoft.AspNetCore.Routing.IParameterPolicy>
<class [Microsoft.AspNetCore.Routing]Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1/Criterion<class [Microsoft.AspNetCore.Routing]Microsoft.AspNetCore.Routing.Tree.OutboundMatch>>
<class [Microsoft.AspNetCore.Routing]Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeNode`1<class [Microsoft.AspNetCore.Routing]Microsoft.AspNetCore.Routing.Tree.OutboundMatch>>
<class [Microsoft.AspNetCore.Server.Kestrel.Core]Microsoft.AspNetCore.Server.Kestrel.Core.Internal.CertificateConfig>
<class [Microsoft.AspNetCore.Server.Kestrel.Core]Microsoft.AspNetCore.Server.Kestrel.Core.Internal.ConfigSectionClone>
<class [Microsoft.AspNetCore.Server.Kestrel.Core]Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Stream>
<class [Microsoft.AspNetCore.Server.Kestrel.Core]Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3PendingStream>
<class [Microsoft.AspNetCore.Server.Kestrel.Core]Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.IHttp3Stream>
<class [Microsoft.AspNetCore.Server.Kestrel.Core]Microsoft.AspNetCore.Server.Kestrel.Core.Internal.SniConfig>
<class [Microsoft.AspNetCore.Server.Kestrel.Core]Microsoft.AspNetCore.Server.Kestrel.Core.Internal.WebTransport.WebTransportSession>
<class [Microsoft.Extensions.Logging]Microsoft.Extensions.Logging.Logger>
<class [System.Linq.Expressions]System.Linq.Expressions.Interpreter.LabelInfo>
<class [System.Linq.Expressions]System.Linq.Expressions.Interpreter.LocalVariable>
<class [System.Linq.Expressions]System.Linq.Expressions.LabelTarget>
<class [System.Private.CoreLib]System.Action`1<class [Microsoft.AspNetCore.Server.Kestrel.Core]Microsoft.AspNetCore.Server.Kestrel.EndpointConfiguration>>
<class [System.Private.CoreLib]System.Collections.Generic.IReadOnlyList`1<class [Microsoft.AspNetCore.Routing]Microsoft.AspNetCore.Routing.Patterns.RoutePatternParameterPolicyReference>>
<class [System.Private.CoreLib]System.Collections.Generic.List`1<class [Microsoft.AspNetCore.Http.Abstractions]Microsoft.AspNetCore.Http.Endpoint>>
<class [System.Private.CoreLib]System.Collections.Generic.List`1<class [Microsoft.AspNetCore.Routing.Abstractions]Microsoft.AspNetCore.Routing.IRouteConstraint>>
<class [System.Private.CoreLib]System.Collections.Generic.List`1<class [Microsoft.AspNetCore.Routing]Microsoft.AspNetCore.Routing.Patterns.RoutePatternParameterPolicyReference>>
<class [System.Private.CoreLib]System.Collections.Generic.List`1<string>>
<class [System.Private.CoreLib]System.Collections.Generic.List`1<valuetype [Microsoft.AspNetCore.Routing]Microsoft.AspNetCore.Routing.Tree.OutboundMatchResult>>
<class [System.Private.CoreLib]System.Globalization.CultureData>
<class [System.Private.CoreLib]System.Globalization.CultureInfo>
<class [System.Private.CoreLib]System.Reflection.Assembly>
<class [System.Private.CoreLib]System.Reflection.Runtime.MethodInfos.CustomMethodInvokerAction>
<class [System.Private.CoreLib]System.Resources.ResourceSet>
<class [System.Private.CoreLib]System.WeakReference`1<class [System.Private.CoreLib]System.Runtime.Loader.AssemblyLoadContext>>
<class [System.Text.Json]System.Text.Json.Serialization.Metadata.JsonTypeInfo/ParameterLookupKey>
<class [System.Text.Json]System.Text.Json.Serialization.Metadata.JsonTypeInfo/ParameterLookupValue>
<class [System.Text.Json]System.Text.Json.Serialization.Metadata.PolymorphicTypeResolver/DerivedJsonTypeInfo>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [Microsoft.AspNetCore.Http.Extensions]Microsoft.AspNetCore.Http.ParameterBindingMethodCache/ParameterLookupKey,class [System.Privat...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Linq.Expressions]System.Linq.Expressions.Expression,class [System.Linq.Expressions]System.Linq.Expressions.Expression/Exten...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Linq.Expressions]System.Linq.Expressions.ParameterExpression,class [System.Linq.Expressions]System.Linq.Expressions.Interpr...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Linq.Expressions]System.Linq.Expressions.ParameterExpression,int32>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Net.Http]System.Net.Http.HttpMethod,int32>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Net.Quic]System.Net.Quic.QuicConnection,class [Microsoft.AspNetCore.Server.Kestrel.Transport.Quic]Microsoft.AspNetCore.Serv...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Net.Sockets]System.Net.Sockets.SafeSocketHandle,class [System.Net.Sockets]System.Net.Sockets.IOControlKeepAlive>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Private.CoreLib]System.__Canon,bool>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Private.CoreLib]System.__Canon,native int>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Private.CoreLib]System.__Canon,valuetype [Microsoft.AspNetCore.Routing]Microsoft.AspNetCore.Routing.DecisionTree.DecisionCri...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Private.CoreLib]System.__Canon,valuetype [Microsoft.Extensions.DependencyInjection]Microsoft.Extensions.DependencyInjection....
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Private.CoreLib]System.__Canon,valuetype [System.Diagnostics.Process]System.Diagnostics.NtProcessManager/ValueId>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Private.CoreLib]System.__Canon,valuetype [System.Private.CoreLib]System.Reflection.NullabilityState>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Private.CoreLib]System.__Canon,valuetype [System.Private.CoreLib]System.ValueTuple`2<class [System.Private.CoreLib]System.__...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Private.CoreLib]System.__Canon,valuetype [System.Security.Cryptography]System.Security.Cryptography.X509Certificates.X509Key...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Private.CoreLib]System.Reflection.Assembly,class [System.Private.CoreLib]System.Runtime.InteropServices.DllImportResolver>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Private.CoreLib]System.Reflection.FieldInfo,class [System.Linq.Expressions]System.Linq.Expressions.Interpreter.Instruction>...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Private.CoreLib]System.Reflection.MemberInfo,valuetype [System.Private.CoreLib]System.Reflection.NullabilityState>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Private.CoreLib]System.Reflection.MethodBase,class [System.Private.CoreLib]System.Reflection.Runtime.MethodInfos.CustomMeth...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Private.CoreLib]System.Reflection.Module,valuetype [System.Private.CoreLib]System.Reflection.NullabilityInfoContext/NotAnno...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Private.CoreLib]System.Reflection.RuntimeAssemblyName,object>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Private.CoreLib]System.Type,class [System.Text.Json]System.Text.Json.Serialization.JsonConverter>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Private.CoreLib]System.Type,string>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Private.CoreLib]System.Type,valuetype [Microsoft.Extensions.DependencyInjection]Microsoft.Extensions.DependencyInjection.Se...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Text.Json]System.Text.Json.JsonSerializerOptions,object>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<class [System.Text.Json]System.Text.Json.Serialization.Metadata.JsonTypeInfo/ParameterLookupKey,class [System.Text.Json]System.Text.Json....
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<int32,class [Microsoft.AspNetCore.Server.Kestrel.Core]Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Stream>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<int32,class [System.Diagnostics.Process]System.Diagnostics.ProcessInfo>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<int32,class [System.Private.CoreLib]System.__Canon>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<int32,int32>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<int32,string>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<int32,valuetype [System.Private.CoreLib]System.ValueTuple`2<class [System.Private.CoreLib]System.__Canon,class [System.Private.CoreLib]Sys...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<int32,valuetype [System.Private.CoreLib]System.ValueTuple`2<string,string>>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<int64,class [Microsoft.AspNetCore.Server.Kestrel.Core]Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3PendingStream>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<int64,class [Microsoft.AspNetCore.Server.Kestrel.Core]Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.IHttp3Stream>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<int64,class [Microsoft.AspNetCore.Server.Kestrel.Core]Microsoft.AspNetCore.Server.Kestrel.Core.Internal.WebTransport.WebTransportSession>...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<int64,class [System.Private.CoreLib]System.__Canon>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<int64,class [System.Private.CoreLib]System.WeakReference`1<class [System.Private.CoreLib]System.Runtime.Loader.AssemblyLoadContext>>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<object,class [Microsoft.AspNetCore.Routing]Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeNode`1<class [Microsoft.AspNetCore.Routi...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<object,class [Microsoft.AspNetCore.Routing]Microsoft.AspNetCore.Routing.Matching.DfaNode>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<object,class [System.Private.CoreLib]System.Threading.Condition>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<object,class [System.Text.Json]System.Text.Json.Serialization.Metadata.PolymorphicTypeResolver/DerivedJsonTypeInfo>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<object,int32>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<object,object>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,bool>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,class [Microsoft.AspNetCore.Http.Abstractions]Microsoft.AspNetCore.Http.Endpoint[]>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,class [Microsoft.AspNetCore.Routing.Abstractions]Microsoft.AspNetCore.Routing.IParameterPolicy>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,class [Microsoft.AspNetCore.Routing]Microsoft.AspNetCore.Routing.Matching.DfaNode>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,class [Microsoft.AspNetCore.Server.Kestrel.Core]Microsoft.AspNetCore.Server.Kestrel.Core.Internal.CertificateConfig>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,class [Microsoft.AspNetCore.Server.Kestrel.Core]Microsoft.AspNetCore.Server.Kestrel.Core.Internal.ConfigSectionClone>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,class [Microsoft.AspNetCore.Server.Kestrel.Core]Microsoft.AspNetCore.Server.Kestrel.Core.Internal.SniConfig>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,class [Microsoft.Extensions.Logging]Microsoft.Extensions.Logging.Logger>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,class [System.Private.CoreLib]System.Action`1<class [Microsoft.AspNetCore.Server.Kestrel.Core]Microsoft.AspNetCore.Server.Kestrel....
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,class [System.Private.CoreLib]System.Collections.Generic.IReadOnlyList`1<class [Microsoft.AspNetCore.Routing]Microsoft.AspNetCore....
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,class [System.Private.CoreLib]System.Collections.Generic.List`1<class [Microsoft.AspNetCore.Routing.Abstractions]Microsoft.AspNetC...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,class [System.Private.CoreLib]System.Collections.Generic.List`1<valuetype [Microsoft.AspNetCore.Routing]Microsoft.AspNetCore.Routi...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,class [System.Private.CoreLib]System.Globalization.CultureData>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,class [System.Private.CoreLib]System.Globalization.CultureInfo>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,class [System.Private.CoreLib]System.Reflection.Assembly>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,class [System.Private.CoreLib]System.Resources.ResourceSet>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,class [System.Private.CoreLib]System.TimeZoneInfo>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,class [System.Private.CoreLib]System.Type>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,native int>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,valuetype [Microsoft.AspNetCore.Routing]Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,valuetype [System.Diagnostics.Process]System.Diagnostics.NtProcessManager/ValueId>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,valuetype [System.Private.CoreLib]System.Resources.ResourceLocator>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,valuetype [System.Private.CoreLib]System.ValueTuple`2<class [System.Text.Json]System.Text.Json.Serialization.Metadata.JsonProperty...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<string,valuetype [System.Security.Cryptography]System.Security.Cryptography.X509Certificates.X509KeyUsageFlags>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<valuetype [Microsoft.AspNetCore.Routing]Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue,class [System.Private.CoreLib]Sy...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<valuetype [Microsoft.AspNetCore.Routing]Microsoft.AspNetCore.Routing.Matching.HttpMethodMatcherPolicy/EdgeKey,class [System.Private.CoreL...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<valuetype [Microsoft.AspNetCore.Routing]Microsoft.AspNetCore.Routing.Matching.HttpMethodMatcherPolicy/EdgeKey,class [System.Private.CoreLi...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<valuetype [Microsoft.Extensions.DependencyInjection]Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceCacheKey,class [System.P...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<valuetype [Microsoft.Extensions.DependencyInjection]Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceCacheKey,object>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<valuetype [System.Net.Security]Interop/SECURITY_STATUS,valuetype [System.Net.Security]System.Net.SecurityStatusPalErrorCode>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<valuetype [System.Net.Security]System.Net.SecurityStatusPalErrorCode,valuetype [System.Net.Security]Interop/SECURITY_STATUS>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<valuetype [System.Private.CoreLib]System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1/ThreadLocalArray<char>[],object>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<valuetype [System.Private.CoreLib]System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1/ThreadLocalArray<int32>[],object>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<valuetype [System.Private.CoreLib]System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1/ThreadLocalArray<object>[],object>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<valuetype [System.Private.CoreLib]System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1/ThreadLocalArray<uint32>[],object>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<valuetype [System.Private.CoreLib]System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1/ThreadLocalArray<uint8>[],object>>
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<valuetype [System.Private.CoreLib]System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1/ThreadLocalArray<valuetype [System.Private.CoreLib...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<valuetype [System.Private.CoreLib]System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1/ThreadLocalArray<valuetype [System.Private.CoreLib...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<valuetype [System.Private.CoreLib]System.ValueTuple`2<class [System.Private.CoreLib]System.__Canon,int32>,valuetype [System.Private.CoreLi...
<valuetype [System.Private.CoreLib]System.Collections.Generic.KeyValuePair`2<valuetype [System.Private.CoreLib]System.ValueTuple`2<string,int32>,valuetype [System.Private.CoreLib]System.ValueTuple`2<class [System.P...

So I would say that special casing the empty enumerator is worth it for KeyValuePair (75% of the uses above).

[jkotas]

main...stephentoub:runtime:emptyenumerator

This saves 56kB in BasicMinimalApi app on top of the 68kB saving from #82751. I think it is worth taking.

[stephentoub]

Ok, I'll wait until yours goes in and then put up a PR for mine.