Add spec document and implement File I/O task migration for multithreaded MSBuild

.config/git-merge-flow-config.jsonc

@@ -28,9 +28,9 @@
         },
         // Automate opening PRs to merge msbuild's vs18.0 (SDK 10.0.1xx) into vs18.3 (SDK 10.0.2xx, VS)
         "vs18.0": {
-            "MergeToBranch": "main" // update to flow through vs18.3 after we fork for release
+            "MergeToBranch": "vs18.3"
         },
-        // MSBuild latest release to main
+        // Automate opening PRs to merge msbuild's vs18.3 (SDK 10.0.2xx) into main
         "vs18.3": {
             "MergeToBranch": "main"
         }

.editorconfig

@@ -415,6 +415,12 @@ dotnet_diagnostic.IDE0301.severity = suggestion
 dotnet_diagnostic.IDE0305.severity = suggestion
 dotnet_diagnostic.IDE0306.severity = suggestion
 
+# Remove unnecessary nullable warning suppression
+dotnet_diagnostic.IDE0370.severity = suggestion
+
+# Remove unnecessary 'unsafe' modifier
+dotnet_diagnostic.IDE0380.severity = suggestion
+
 # Temporarily disable SA1010 "Opening square brackets should not be preceded by a space" until https://github.com/DotNetAnalyzers/StyleCopAnalyzers/issues/3687 is fixed
 dotnet_diagnostic.SA1010.severity = none
 

.github/copilot-instructions.md

@@ -11,8 +11,90 @@ New files should use nullable types but don't refactor aggressively existing cod
 
 Generate tests for new codepaths, and add tests for any bugs you fix. Use the existing test framework, which is xUnit with Shouldly assertions. Use Shouldly assertions for all assertions in modified code, even if the file is predominantly using xUnit assertions.
 
+When making changes, check if related documentation exists in the `documentation/` folder (including `documentation/specs/`) and update it to reflect your changes. Keep documentation in sync with code changes, especially for telemetry, APIs, and architectural decisions.
+
 Always reference these instructions first and fallback to search or bash commands only when you encounter unexpected information that does not match the info here.
 
+## Performance Best Practices
+
+MSBuild is performance-critical infrastructure. Follow these patterns:
+
+### Switch Expressions for Dispatch Logic
+Use tuple switch expressions for multi-condition dispatch instead of if-else chains:
+```csharp
+// GOOD: Clean, O(1) dispatch
+return (c0, c1) switch
+{
+    ('C', 'S') => Category.CSharp,
+    ('F', 'S') => Category.FSharp,
+    ('V', 'B') when value.Length >= 3 && value[2] == 'C' => Category.VB,
+    _ => Category.Other
+};
+
+// AVOID: Verbose if-else chains
+if (c0 == 'C' && c1 == 'S') return Category.CSharp;
+else if (c0 == 'F' && c1 == 'S') return Category.FSharp;
+// ...
+```
+
+### Range Pattern Matching
+Use range patterns for numeric categorization:
+```csharp
+// GOOD: Clear and efficient
+return errorNumber switch
+{
+    >= 3001 and <= 3999 => Category.Tasks,
+    >= 4001 and <= 4099 => Category.General,
+    >= 4100 and <= 4199 => Category.Evaluation,
+    _ => Category.Other
+};
+```
+
+### String Comparisons
+- Use `StringComparer.OrdinalIgnoreCase` for case-insensitive HashSets/Dictionaries when the source data may vary in casing
+- Use `char.ToUpperInvariant()` for single-character comparisons
+- Use `ReadOnlySpan<char>` and `Slice()` to avoid string allocations when parsing substrings
+- Use `int.TryParse(span, out var result)` on .NET Core+ for allocation-free parsing
+
+### Inlining
+Mark small, hot-path methods with `[MethodImpl(MethodImplOptions.AggressiveInlining)]`:
+```csharp
+[MethodImpl(MethodImplOptions.AggressiveInlining)]
+private static bool IsCompilerPrefix(string value) => ...
+```
+
+### Conditional Compilation for Framework Differences
+Use `#if NET` for APIs that differ between .NET Framework and .NET Core:
+```csharp
+#if NET
+    return int.TryParse(span, out errorNumber);
+#else
+    return int.TryParse(span.ToString(), out errorNumber);
+#endif
+```
+
+### Immutable Collections
+Choose the right immutable collection type based on usage pattern:
+
+**Build once, read many times** (most common in MSBuild):
+- Use `ImmutableArray<T>` instead of `ImmutableList<T>` - significantly faster for read access
+- Use `FrozenDictionary<TKey, TValue>` instead of `ImmutableDictionary<TKey, TValue>` - optimized for read-heavy scenarios
+
+**Build incrementally over time** (adding items one by one):
+- Use `ImmutableList<T>` and `ImmutableDictionary<TKey, TValue>` - designed for efficient `Add` operations returning new collections
+
+```csharp
+// GOOD: Build once from LINQ, then read many times
+ImmutableArray<string> items = source.Select(x => x.Name).ToImmutableArray();
+FrozenDictionary<string, int> lookup = pairs.ToFrozenDictionary(x => x.Key, x => x.Value);
+
+// AVOID for read-heavy scenarios:
+ImmutableList<string> items = source.Select(x => x.Name).ToImmutableList();
+ImmutableDictionary<string, int> lookup = pairs.ToImmutableDictionary(x => x.Key, x => x.Value);
+```
+
+Note: `ImmutableArray<T>` is a value type. Use `IsDefault` property to check for uninitialized arrays, or use nullable `ImmutableArray<T>?` with `.Value` to unwrap.
+
 ## Working Effectively
 
 #### Bootstrap and Build the Repository

.vsts-dotnet-ci.yml

@@ -20,6 +20,14 @@ jobs:
   steps:
   - powershell: |
       $versionsFile = "eng/Versions.props"
+
+      [xml]$xml = Get-Content $versionsFile
+      $finalVersionKind = $xml.Project.PropertyGroup.DotNetFinalVersionKind
+      if ($finalVersionKind -ne 'release') {
+        Write-Host "Since it is not released, skip the version bump check.";
+        return
+      }
+
       $changedFiles = git diff --name-only HEAD HEAD~1
       $changedVersionsFile = $changedFiles | Where-Object { $_ -eq $versionsFile }
       $isInitialCommit = $false

Directory.Packages.props

@@ -45,11 +45,10 @@
     <PackageVersion Include="System.Threading.Channels" Version="$(SystemThreadingChannelsVersion)" />
     <PackageVersion Include="System.Threading.Tasks.Dataflow" Version="$(SystemThreadingTasksDataflowVersion)" />
     <PackageVersion Include="System.Threading.Tasks.Extensions" Version="$(SystemThreadingTasksExtensionsVersion)" />
+    <PackageVersion Include="System.ValueTuple" Version="$(SystemValueTupleVersion)" />
     <PackageVersion Include="xunit.console" Version="$(XUnitVersion)" />
-    <PackageVersion Include="Microsoft.VisualStudio.OpenTelemetry.Collector" Version="$(MicrosoftVisualStudioOpenTelemetryVersion)" />
-    <PackageVersion Include="Microsoft.VisualStudio.OpenTelemetry.ClientExtensions" Version="$(MicrosoftVisualStudioOpenTelemetryVersion)" />
-
-    <!-- Microsoft.VisualStudio.SolutionPersistence is maintained in eng/dependabot/Directory.Packages.props -->
+    <PackageVersion Include="Microsoft.VisualStudio.Telemetry" Version="$(MicrosoftVisualStudioTelemetryVersion)" />
+    <!-- Microsoft.VisualStudio.SolutionPersistence is maintained in eng/dependabot/Packages.props -->
   </ItemGroup>
 
 </Project>

NuGet.config

@@ -16,13 +16,11 @@
     <add key="dotnet9-transport" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet9-transport/nuget/v3/index.json" />
     <add key="dotnet10" value="https://dnceng.pkgs.visualstudio.com/public/_packaging/dotnet10/nuget/v3/index.json" />
     <add key="dotnet10-transport" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet10-transport/nuget/v3/index.json" />
+    <add key="dotnet11" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet11/nuget/v3/index.json" />
+    <add key="dotnet11-transport" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet11-transport/nuget/v3/index.json" />
     <add key="vs-impl" value="https://pkgs.dev.azure.com/azure-public/vside/_packaging/vs-impl/nuget/v3/index.json" />
   </packageSources>
-  <!-- We don't want other than VS OpenTelemetry libraries from vs-impl -->
   <packageSourceMapping>
-    <packageSource key="vs-impl">
-      <package pattern="Microsoft.VisualStudio.OpenTelemetry*" />
-    </packageSource>
     <packageSource key="arcade">
       <package pattern="*" />
     </packageSource>
@@ -53,6 +51,12 @@
     <packageSource key="dotnet10-transport">
       <package pattern="*" />
     </packageSource>
+    <packageSource key="dotnet11">
+      <package pattern="*" />
+    </packageSource>
+    <packageSource key="dotnet11-transport">
+      <package pattern="*" />
+    </packageSource>
   </packageSourceMapping>
   <disabledPackageSources>
     <!--Begin: Package sources managed by Dependency Flow automation. Do not edit the sources below.-->

THIRDPARTYNOTICES.txt

@@ -43,33 +43,3 @@ Unless required by applicable law or agreed to in writing, software distributed
 under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
 CONDITIONS OF ANY KIND, either express or implied. See the License for the specific
 language governing permissions and limitations under the License.
-
--------------------------------
-
-Notice for OpenTelemetry .NET
--------------------------------
-MSBuild.exe is distributed with OpenTelemetry .NET binaries.
-
-Copyright (c) OpenTelemetry Authors
-Source:       https://github.com/open-telemetry/opentelemetry-dotnet
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software distributed
-under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
-CONDITIONS OF ANY KIND, either express or implied. See the License for the specific
-language governing permissions and limitations under the License.
-
--------------------------------
-
-Notice for Microsoft.VisualStudio.OpenTelemetry.*
--------------------------------
-MSBuild.exe is distributed with Microsoft.VisualStudio.OpenTelemetry.* binaries.
-
-Project:      Microsoft.VisualStudio.OpenTelemetry
-Copyright:    (c) Microsoft Corporation
-License:      https://visualstudio.microsoft.com/license-terms/mt736442/

azure-pipelines/.vsts-dotnet-build-jobs.yml

@@ -29,7 +29,10 @@ jobs:
     - output: artifactsDrop
       sourcePath: '$(Build.SourcesDirectory)\artifacts\official\OptProf\$(BuildConfiguration)\Data'
       dropServiceURI: 'https://devdiv.artifacts.visualstudio.com'
-      dropMetadataContainerName: 'ProfilingInputs/DevDiv/$(Build.Repository.Name)/$(Build.SourceBranchName)/$(Build.BuildNumber)'
+      buildNumber: 'ProfilingInputs/DevDiv/$(Build.Repository.Name)/$(Build.SourceBranchName)/$(Build.BuildNumber)'
+      toLowerCase: false
+      usePat: true
+      dropMetadataContainerName: 'DropMetadata-OptProf'
       condition: and(succeeded(), ${{ parameters.enableOptProf }})
 
     # Publish bootstrapper info for OptProf data collection run to consume

azure-pipelines/vs-insertion-experimental.yml

@@ -24,6 +24,8 @@ parameters:
     displayName: 'Insertion Target Branch (select for manual insertion)'
     values:
       - main
+      - rel/d18.4
+      - rel/d18.3
       - rel/d18.0
       - rel/d17.14
       - rel/d17.13

azure-pipelines/vs-insertion.yml

@@ -44,6 +44,7 @@ parameters:
     values:
       - auto
       - main
+      - rel/d18.4
       - rel/d18.3
       - rel/d18.0
       - rel/d17.14
@@ -67,7 +68,9 @@ parameters:
 variables:
   # `auto` should work every time and selecting a branch in parameters is likely to fail due to incompatible versions in MSBuild and VS
   - name: AutoInsertTargetBranch
-    ${{ if eq(variables['Build.SourceBranchName'], 'vs18.3') }}:
+    ${{ if eq(variables['Build.SourceBranchName'], 'vs18.4') }}:
+      value: 'rel/d18.4'
+    ${{ elseif eq(variables['Build.SourceBranchName'], 'vs18.3') }}:
       value: 'rel/d18.3'
     ${{ elseif eq(variables['Build.SourceBranchName'], 'vs18.0') }}:
       value: 'rel/d18.0'

documentation/High-level-overview.md

@@ -149,6 +149,15 @@ TaskHost can be opted-in via `TaskFactory="TaskHostFactory"` in the [`UsingTask`
 - If a task's source code is in the same repository that is being built, and the repository's build needs to use that task during the build process. Using a Task Host makes sure the DLLs are not locked at the end of the build (as MSBuild uses long living worker nodes that survives single build execution)
 - As an isolation mechanism - separating the execution from the engine execution process.
 
+When `TaskHostFactory` is specified as the task factory, the task always runs out-of-process and short lived. See the below matrix:
+
+| Does TaskHost match executing MSBuild Runtime? | Is TaskHostFactory requested for the Task? | Expected task execution type |
+| :-: | :-: | --- |
+| ✅ | :x: | in-process execution |
+| ✅ | ✅ | short-lived out-of-proc execution |
+| :x: | ✅ | short-lived out-of-proc execution |
+| :x: | :x: | long-lived out-of-proc execution |
+
 ## Caches
 ### Project result cache
 The project Result Cache refers to the cache used by the scheduler that keeps the build results of already executed project. The result of a target is success, failure, and a list of items that succeeded. Beyond that, the `Returns` and `Outputs` attributes from targets are also serialized with the build result, as to be used by other targets for their execution.