Merge pull request #17 from joperezr/Utf8String-ingestion

Dotnet runtime master ingestion into Utf8String experiment
dotnet · Jul 17, 2020 · 82c3b4c · 82c3b4c
2 parents 703c516 + cd23211
commit 82c3b4c
Show file tree

Hide file tree

Showing 1,175 changed files with 97,677 additions and 11,926 deletions.
diff --git a/THIRD-PARTY-NOTICES.TXT b/THIRD-PARTY-NOTICES.TXT
@@ -860,3 +860,28 @@ The above copyright notice and this permission notice shall be included in all c
 
 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 
+License notice for vectorized sorting code
+------------------------------------------
+
+MIT License
+
+Copyright (c) 2020 Dan Shechter
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
diff --git a/docs/deep-dive-blog-posts.md b/docs/deep-dive-blog-posts.md
@@ -6,6 +6,7 @@
 - [Performance improvements in .NET Core 2.0](https://devblogs.microsoft.com/dotnet/performance-improvements-in-net-core/)
 - [Performance improvements in .NET Core 2.1](https://devblogs.microsoft.com/dotnet/performance-improvements-in-net-core-2-1/)
 - [Performance improvements in .NET Core 3.0](https://devblogs.microsoft.com/dotnet/performance-improvements-in-net-core-3-0/)
+- [Performance improvements in .NET 5](https://devblogs.microsoft.com/dotnet/performance-improvements-in-net-5/)
 
 
 ### Posts that take a high-level look at the entire source:

diff --git a/docs/project/list-of-obsoletions.md b/docs/project/list-of-obsoletions.md
@@ -15,3 +15,4 @@ Currently the identifiers `MSLIB0001` through `MSLIB0999` are carved out for obs
 | :--------------- | :---------- |
 |  __`MSLIB0001`__ | The UTF-7 encoding is insecure and should not be used. Consider using UTF-8 instead. |
 |  __`MSLIB0002`__ | `PrincipalPermissionAttribute` is not honored by the runtime and must not be used. |
+|  __`MSLIB0003`__ | `BinaryFormatter` serialization is obsolete and should not be used. See https://aka.ms/binaryformatter for recommended alternatives. |
diff --git a/docs/workflow/trimming/feature-switches.md b/docs/workflow/trimming/feature-switches.md
@@ -0,0 +1,57 @@
+# Libraries Feature Switches
+
+Starting with .NET 5 there are several [feature-switches](https://github.com/dotnet/designs/blob/master/accepted/2020/feature-switch.md) available which
+can be used to control the size of the final binary. They are available in all
+configurations but their defaults might vary as any SDK can set the defaults differently.
+
+## Available Feature Switches
+
+| MSBuild Property Name | AppContext Setting | Description |
+|-|-|-|
+| DebuggerSupport | System.Diagnostics.Debugger.IsSupported | Any dependency that enables better debugging experience to be trimmed when set to false |
+| EnableUnsafeUTF7Encoding | System.Text.Encoding.EnableUnsafeUTF7Encoding | Insecure UTF-7 encoding is trimmed when set to false |
+| EnableUnsafeBinaryFormatterSerialization | System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization | BinaryFormatter serialization support is trimmed when set to false |
+| EventSourceSupport | System.Diagnostics.Tracing.EventSource.IsSupported | Any EventSource related code or logic is trimmed when set to false |
+| InvariantGlobalization | System.Globalization.Invariant | All globalization specific code and data is trimmed when set to true |
+| UseSystemResourceKeys | System.Resources.UseSystemResourceKeys |  Any localizable resources for system assemblies is trimmed when set to true |
+| - | System.Net.Http.EnableActivityPropagation | Any dependency related to diagnostics support for System.Net.Http is trimmed when set to false |
+
+Any feature-switch which defines property can be set in csproj file or
+on the command line as any other MSBuild property. Those without predefined property name
+the value can be set with following XML tag in csproj file.
+
+```xml
+<RuntimeHostConfigurationOption Include="<AppContext-Setting>"
+                                Value="false"
+                                Trim="true" />
+```
+
+## Adding New Feature Switch
+
+The primary goal of features switches is to produce smaller output by removing code which is
+unreachable under feature condition. The typical approach is to introduce static bool like
+property which is used to guard the dependencies which can be trimmed when the value is flipped.
+Ideally, the static property should be located in type which does not have any static constructor
+logic. Once you are done with the code changes following steps connects the code with trimming
+settings.
+
+Add XML settings for the features switch to assembly substitution. It's usually located in
+`src/ILLink/ILLink.Substitutions.xml` file for each library. The example of the syntax used to control
+`EnableUnsafeUTF7Encoding` property is following.
+
+```xml
+<method signature="System.Boolean get_EnableUnsafeUTF7Encoding()" body="stub" value="false" feature="System.Text.Encoding.EnableUnsafeUTF7Encoding" featurevalue="false" />
+```
+
+Add MSBuild integration by adding new RuntimeHostConfigurationOption entry. The file is located in
+[Microsoft.NET.Sdk.targets](https://github.com/dotnet/sdk/blob/33ce6234e6bf45bce16f610c441679252d309189/src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.Sdk.targets#L348-L401) file and includes all
+other public feature-switches. You can add a new one by simply adding a new XML tag
+
+```xml
+<RuntimeHostConfigurationOption Include="<AppContext-Setting>"
+            Condition="'$(<msbuild-property-name>)' != ''"
+            Value="$(<msbuild-property-name>)"
+            Trim="true" />
+```
+
+Please don't forget to update the table with available features-switches when you are done.
diff --git a/eng/Analyzers.props b/eng/Analyzers.props
@@ -6,7 +6,7 @@
   </PropertyGroup>
   <ItemGroup Condition="'$(EnableAnalyzers)' == 'true'">
     <PackageReference Include="Microsoft.DotNet.CodeAnalysis" Version="$(MicrosoftDotNetCodeAnalysisVersion)" PrivateAssets="all" IsImplicitlyDefined="true" />
-    <PackageReference Include="Microsoft.CodeAnalysis.NetAnalyzers" Version="3.3.0-beta1.20327.1" PrivateAssets="all" />
+    <PackageReference Include="Microsoft.CodeAnalysis.NetAnalyzers" Version="3.3.0-beta1.20355.1" PrivateAssets="all" />
     <PackageReference Include="StyleCop.Analyzers" Version="1.2.0-beta.164" PrivateAssets="all" />
   </ItemGroup>
 </Project>
diff --git a/eng/CodeAnalysis.ruleset b/eng/CodeAnalysis.ruleset
@@ -122,7 +122,7 @@
       <Rule Id="CA2013" Action="Warning" />          <!-- Do not use ReferenceEquals with value types -->
       <Rule Id="CA2014" Action="Warning" />          <!-- Do not use stackalloc in loops. -->
       <Rule Id="CA2015" Action="Warning" />          <!-- Do not define finalizers for types derived from MemoryManager<T> -->
-      <Rule Id="CA2016" Action="Info" />             <!-- Forward the 'CancellationToken' parameter to methods that take one -->
+      <Rule Id="CA2016" Action="Warning" />          <!-- Forward the 'CancellationToken' parameter to methods that take one -->
       <Rule Id="CA2100" Action="None" />             <!-- Review SQL queries for security vulnerabilities -->
       <Rule Id="CA2101" Action="None" />             <!-- Specify marshaling for P/Invoke string arguments -->
       <Rule Id="CA2109" Action="None" />             <!-- Review visible event handlers -->

diff --git a/eng/Configurations.props b/eng/Configurations.props
@@ -31,6 +31,7 @@
     <NetCoreAppCurrentBrandName>.NET $(NetCoreAppCurrentVersion)</NetCoreAppCurrentBrandName>
 
     <NetFrameworkCurrent>net472</NetFrameworkCurrent>
+    <MinimiumSupportedWindowsPlatform>WINDOWS7.0</MinimiumSupportedWindowsPlatform>
   </PropertyGroup>
 
   <!-- Honor the generic RuntimeConfiguration property. -->

diff --git a/eng/Version.Details.xml b/eng/Version.Details.xml
@@ -4,63 +4,67 @@
       <Uri>https://github.com/dotnet/standard</Uri>
       <Sha>cfe95a23647c7de1fe1a349343115bd7720d6949</Sha>
     </Dependency>
+    <Dependency Name="Microsoft.NETCore.Runtime.ICU.Transport" Version="5.0.0-preview.8.20365.1">
+      <Uri>https://github.com/dotnet/icu</Uri>
+      <Sha>7247fa0d9e8faee2cceee6f04856b2c447f41bca</Sha>
+    </Dependency>
   </ProductDependencies>
   <ToolsetDependencies>
-    <Dependency Name="Microsoft.DotNet.Arcade.Sdk" Version="5.0.0-beta.20330.3">
+    <Dependency Name="Microsoft.DotNet.Arcade.Sdk" Version="5.0.0-beta.20364.3">
       <Uri>https://github.com/dotnet/arcade</Uri>
-      <Sha>243cc92161ad44c2a07464425892daee19121c99</Sha>
+      <Sha>ff5d4b6c8dbdaeacb6e6159d3f8185118dffd915</Sha>
     </Dependency>
-    <Dependency Name="Microsoft.DotNet.Helix.Sdk" Version="5.0.0-beta.20330.3">
+    <Dependency Name="Microsoft.DotNet.Helix.Sdk" Version="5.0.0-beta.20364.3">
       <Uri>https://github.com/dotnet/arcade</Uri>
-      <Sha>243cc92161ad44c2a07464425892daee19121c99</Sha>
+      <Sha>ff5d4b6c8dbdaeacb6e6159d3f8185118dffd915</Sha>
     </Dependency>
-    <Dependency Name="Microsoft.DotNet.ApiCompat" Version="5.0.0-beta.20330.3">
+    <Dependency Name="Microsoft.DotNet.ApiCompat" Version="5.0.0-beta.20364.3">
       <Uri>https://github.com/dotnet/arcade</Uri>
-      <Sha>243cc92161ad44c2a07464425892daee19121c99</Sha>
+      <Sha>ff5d4b6c8dbdaeacb6e6159d3f8185118dffd915</Sha>
     </Dependency>
-    <Dependency Name="Microsoft.DotNet.GenAPI" Version="5.0.0-beta.20330.3">
+    <Dependency Name="Microsoft.DotNet.GenAPI" Version="5.0.0-beta.20364.3">
       <Uri>https://github.com/dotnet/arcade</Uri>
-      <Sha>243cc92161ad44c2a07464425892daee19121c99</Sha>
+      <Sha>ff5d4b6c8dbdaeacb6e6159d3f8185118dffd915</Sha>
     </Dependency>
-    <Dependency Name="Microsoft.DotNet.GenFacades" Version="5.0.0-beta.20330.3">
+    <Dependency Name="Microsoft.DotNet.GenFacades" Version="5.0.0-beta.20364.3">
       <Uri>https://github.com/dotnet/arcade</Uri>
-      <Sha>243cc92161ad44c2a07464425892daee19121c99</Sha>
+      <Sha>ff5d4b6c8dbdaeacb6e6159d3f8185118dffd915</Sha>
     </Dependency>
-    <Dependency Name="Microsoft.DotNet.XUnitExtensions" Version="5.0.0-beta.20330.3">
+    <Dependency Name="Microsoft.DotNet.XUnitExtensions" Version="5.0.0-beta.20364.3">
       <Uri>https://github.com/dotnet/arcade</Uri>
-      <Sha>243cc92161ad44c2a07464425892daee19121c99</Sha>
+      <Sha>ff5d4b6c8dbdaeacb6e6159d3f8185118dffd915</Sha>
     </Dependency>
-    <Dependency Name="Microsoft.DotNet.XUnitConsoleRunner" Version="2.5.1-beta.20330.3">
+    <Dependency Name="Microsoft.DotNet.XUnitConsoleRunner" Version="2.5.1-beta.20364.3">
       <Uri>https://github.com/dotnet/arcade</Uri>
-      <Sha>243cc92161ad44c2a07464425892daee19121c99</Sha>
+      <Sha>ff5d4b6c8dbdaeacb6e6159d3f8185118dffd915</Sha>
     </Dependency>
-    <Dependency Name="Microsoft.DotNet.Build.Tasks.Packaging" Version="5.0.0-beta.20330.3">
+    <Dependency Name="Microsoft.DotNet.Build.Tasks.Packaging" Version="5.0.0-beta.20364.3">
       <Uri>https://github.com/dotnet/arcade</Uri>
-      <Sha>243cc92161ad44c2a07464425892daee19121c99</Sha>
+      <Sha>ff5d4b6c8dbdaeacb6e6159d3f8185118dffd915</Sha>
     </Dependency>
-    <Dependency Name="Microsoft.DotNet.CodeAnalysis" Version="5.0.0-beta.20330.3">
+    <Dependency Name="Microsoft.DotNet.CodeAnalysis" Version="5.0.0-beta.20364.3">
       <Uri>https://github.com/dotnet/arcade</Uri>
-      <Sha>243cc92161ad44c2a07464425892daee19121c99</Sha>
+      <Sha>ff5d4b6c8dbdaeacb6e6159d3f8185118dffd915</Sha>
     </Dependency>
-    <Dependency Name="Microsoft.DotNet.Build.Tasks.TargetFramework.Sdk" Version="5.0.0-beta.20330.3">
+    <Dependency Name="Microsoft.DotNet.Build.Tasks.TargetFramework.Sdk" Version="5.0.0-beta.20364.3">
       <Uri>https://github.com/dotnet/arcade</Uri>
-      <Sha>243cc92161ad44c2a07464425892daee19121c99</Sha>
+      <Sha>ff5d4b6c8dbdaeacb6e6159d3f8185118dffd915</Sha>
     </Dependency>
-    <Dependency Name="Microsoft.DotNet.RemoteExecutor" Version="5.0.0-beta.20330.3">
+    <Dependency Name="Microsoft.DotNet.RemoteExecutor" Version="5.0.0-beta.20364.3">
       <Uri>https://github.com/dotnet/arcade</Uri>
-      <Sha>243cc92161ad44c2a07464425892daee19121c99</Sha>
+      <Sha>ff5d4b6c8dbdaeacb6e6159d3f8185118dffd915</Sha>
     </Dependency>
-    <Dependency Name="Microsoft.DotNet.Build.Tasks.Feed" Version="5.0.0-beta.20330.3">
+    <Dependency Name="Microsoft.DotNet.Build.Tasks.Feed" Version="5.0.0-beta.20364.3">
       <Uri>https://github.com/dotnet/arcade</Uri>
-      <Sha>243cc92161ad44c2a07464425892daee19121c99</Sha>
+      <Sha>ff5d4b6c8dbdaeacb6e6159d3f8185118dffd915</Sha>
     </Dependency>
-    <Dependency Name="Microsoft.DotNet.VersionTools.Tasks" Version="5.0.0-beta.20330.3">
+    <Dependency Name="Microsoft.DotNet.VersionTools.Tasks" Version="5.0.0-beta.20364.3">
       <Uri>https://github.com/dotnet/arcade</Uri>
-      <Sha>243cc92161ad44c2a07464425892daee19121c99</Sha>
+      <Sha>ff5d4b6c8dbdaeacb6e6159d3f8185118dffd915</Sha>
     </Dependency>
-    <Dependency Name="Microsoft.DotNet.Build.Tasks.SharedFramework.Sdk" Version="5.0.0-beta.20330.3">
+    <Dependency Name="Microsoft.DotNet.Build.Tasks.SharedFramework.Sdk" Version="5.0.0-beta.20364.3">
       <Uri>https://github.com/dotnet/arcade</Uri>
-      <Sha>243cc92161ad44c2a07464425892daee19121c99</Sha>
+      <Sha>ff5d4b6c8dbdaeacb6e6159d3f8185118dffd915</Sha>
     </Dependency>
     <Dependency Name="optimization.windows_nt-x64.IBC.CoreFx" Version="99.99.99-master-20190716.1">
       <Uri>https://dev.azure.com/dnceng/internal/_git/dotnet-optimization</Uri>
@@ -86,41 +90,37 @@
       <Uri>https://github.com/microsoft/vstest</Uri>
       <Sha>b9296fc900e2f2d717d507b4ee2a4306521796d4</Sha>
     </Dependency>
-    <Dependency Name="System.ComponentModel.TypeConverter.TestData" Version="5.0.0-beta.20319.2">
+    <Dependency Name="System.ComponentModel.TypeConverter.TestData" Version="5.0.0-beta.20360.1">
       <Uri>https://github.com/dotnet/runtime-assets</Uri>
-      <Sha>629993236116221fba87fe1de6d7893dd02c3722</Sha>
+      <Sha>b34a70799faa67f13c2e72852e4f3af463ff4d6c</Sha>
     </Dependency>
-    <Dependency Name="System.Drawing.Common.TestData" Version="5.0.0-beta.20319.2">
+    <Dependency Name="System.Drawing.Common.TestData" Version="5.0.0-beta.20360.1">
       <Uri>https://github.com/dotnet/runtime-assets</Uri>
-      <Sha>629993236116221fba87fe1de6d7893dd02c3722</Sha>
+      <Sha>b34a70799faa67f13c2e72852e4f3af463ff4d6c</Sha>
     </Dependency>
-    <Dependency Name="System.IO.Compression.TestData" Version="5.0.0-beta.20319.2">
+    <Dependency Name="System.IO.Compression.TestData" Version="5.0.0-beta.20360.1">
       <Uri>https://github.com/dotnet/runtime-assets</Uri>
-      <Sha>629993236116221fba87fe1de6d7893dd02c3722</Sha>
+      <Sha>b34a70799faa67f13c2e72852e4f3af463ff4d6c</Sha>
     </Dependency>
-    <Dependency Name="System.IO.Packaging.TestData" Version="5.0.0-beta.20319.2">
+    <Dependency Name="System.IO.Packaging.TestData" Version="5.0.0-beta.20360.1">
       <Uri>https://github.com/dotnet/runtime-assets</Uri>
-      <Sha>629993236116221fba87fe1de6d7893dd02c3722</Sha>
+      <Sha>b34a70799faa67f13c2e72852e4f3af463ff4d6c</Sha>
     </Dependency>
-    <Dependency Name="System.Net.TestData" Version="5.0.0-beta.20319.2">
+    <Dependency Name="System.Net.TestData" Version="5.0.0-beta.20360.1">
       <Uri>https://github.com/dotnet/runtime-assets</Uri>
-      <Sha>629993236116221fba87fe1de6d7893dd02c3722</Sha>
+      <Sha>b34a70799faa67f13c2e72852e4f3af463ff4d6c</Sha>
     </Dependency>
-    <Dependency Name="System.Private.Runtime.UnicodeData" Version="5.0.0-beta.20319.2">
+    <Dependency Name="System.Private.Runtime.UnicodeData" Version="5.0.0-beta.20360.1">
       <Uri>https://github.com/dotnet/runtime-assets</Uri>
-      <Sha>629993236116221fba87fe1de6d7893dd02c3722</Sha>
+      <Sha>b34a70799faa67f13c2e72852e4f3af463ff4d6c</Sha>
     </Dependency>
-    <Dependency Name="System.Security.Cryptography.X509Certificates.TestData" Version="5.0.0-beta.20319.2">
+    <Dependency Name="System.Security.Cryptography.X509Certificates.TestData" Version="5.0.0-beta.20360.1">
       <Uri>https://github.com/dotnet/runtime-assets</Uri>
-      <Sha>629993236116221fba87fe1de6d7893dd02c3722</Sha>
+      <Sha>b34a70799faa67f13c2e72852e4f3af463ff4d6c</Sha>
     </Dependency>
-    <Dependency Name="System.Windows.Extensions.TestData" Version="5.0.0-beta.20319.2">
+    <Dependency Name="System.Windows.Extensions.TestData" Version="5.0.0-beta.20360.1">
       <Uri>https://github.com/dotnet/runtime-assets</Uri>
-      <Sha>629993236116221fba87fe1de6d7893dd02c3722</Sha>
-    </Dependency>
-    <Dependency Name="Microsoft.NETCore.Runtime.ICU.Transport" Version="5.0.0-preview.8.20359.5">
-      <Uri>https://github.com/dotnet/icu</Uri>
-      <Sha />
+      <Sha>b34a70799faa67f13c2e72852e4f3af463ff4d6c</Sha>
     </Dependency>
     <Dependency Name="runtime.linux-arm64.Microsoft.NETCore.Runtime.Mono.LLVM.Sdk" Version="9.0.1-alpha.1.20356.1">
       <Uri>https://github.com/dotnet/llvm-project</Uri>
@@ -182,9 +182,9 @@
       <Uri>https://github.com/dotnet/runtime</Uri>
       <Sha>0375524a91a47ca4db3ee1be548f74bab7e26e76</Sha>
     </Dependency>
-    <Dependency Name="Microsoft.NET.ILLink.Tasks" Version="5.0.0-preview.3.20360.3">
+    <Dependency Name="Microsoft.NET.ILLink.Tasks" Version="5.0.0-preview.3.20363.5">
       <Uri>https://github.com/mono/linker</Uri>
-      <Sha>095f30a37d740e5166d71ab2d2157c5bb2041efa</Sha>
+      <Sha>e76321851c05c5016df3d1243885c02e875b9912</Sha>
     </Dependency>
     <Dependency Name="Microsoft.DotNet.XHarness.TestRunners.Xunit" Version="1.0.0-prerelease.20352.3">
       <Uri>https://github.com/dotnet/xharness</Uri>