Enable -mt in PR CIs with 2 stages

[JanProvaznik]

Closes #13602 (PR-CI portion). Turns on /mt in the bootstrapped stage 2 of every existing PR-CI job that already runs a 2-stage build with the freshly built MSBuild.

The original PR also touched .vsts-dotnet.yml (official build) and .vsts-dotnet-exp-perf.yml (experimental insertion) and the cibuild.cmd-based FullReleaseOnWindows job; those were reverted because they all run a single MSBuild invocation against the agent's stable VS MSBuild (no bootstrap re-invocation), so adding /mt there would test whatever MT support the pre-installed MSBuild has, not the freshly built one. Converting those to true 2-stage flows is a separate, larger change. The previously stacked PR #13679 was closed for the same reason.

What this PR does

Job in .vsts-dotnet-ci.yml	Stage / build invocation	Change
BootstrapMSBuildOnFullFrameworkWindows (Windows Full)	bootstrapped stage 2	append -stage2Properties /mt
BootstrapMSBuildOnCoreWindows (Windows Core)	bootstrapped stage 2	append -stage2Properties /mt
CoreBootstrappedOnLinux (Linux Core)	bootstrapped stage 2	append --stage2Properties '/mt'
CoreOnMac (macOS Core)	bootstrapped stage 2	append --stage2Properties '/mt'
BootstrapMSBuildWithMTMode (Linux dedicated MT + ThreadSafeTaskAnalyzer)	bootstrapped stage 2	kept as-is — still the only job that runs /p:BuildAnalyzer=true
FullReleaseOnWindows (single-stage cibuild.cmd)	n/a	untouched — uses agent's stable VS MSBuild
Source-Build template	n/a	untouched

Tests still run in every job that ran them before. The dedicated BootstrapMSBuildWithMTMode keeps --skipTests (as on main).

Files changed

eng/cibuild_bootstrapped_msbuild.ps1

• Add -stage2Properties parameter (mirrors --stage2Properties in the bash script). Lets callers pass /mt to stage 2 only, not to the SDK MSBuild used in stage 1.
• Add -skipTests switch (mirrors --skipTests).
• Three-way branch (mirrors cibuild_bootstrapped_msbuild.sh):
  • -onlyDocChanged → /p:CreateBootstrap=false (bootstrap not created, downstream doesn't need it).
  • -skipTests → bootstrap is created (downstream may consume it), tests omitted.
  • default → bootstrap created, tests run.
• Switch to dotnet.exe msbuild (SDK CLI verb) instead of dotnet.exe exec MSBuild.dll. The CLI sets DOTNET_HOST_PATH for the MSBuild process automatically, which /mt needs so the apphost-based child task host (NodeProviderOutOfProcTaskHost.ResolveAppHostOrFallback) can locate the runtime. Removes the Versions.props lookup for the bootstrap SDK's MSBuild.dll path.
• Fix /mt argument splatting: "/mt" -split '\s+' | Where-Object { $_ } returns a string (not string[]) when there is exactly one token, and & cmd @stringVar then splats the string's characters one per arg — so /mt reached MSBuild as /, m, t. Wrapping with @() keeps the result an array for 0/1/N tokens. This was caught locally; would have failed CI.

.vsts-dotnet-ci.yml

• Append /mt to every stage 2 invocation listed above. Linux MT-with-analyzer job unchanged.

Switched to dotnet msbuild (per Rainer's suggestion)

The script now runs the bootstrap as dotnet.exe msbuild (the SDK CLI verb) rather than dotnet exec MSBuild.dll. The CLI sets DOTNET_HOST_PATH for the MSBuild process automatically, so the explicit env-var export is gone, along with the Versions.props lookup for the bootstrap SDK's MSBuild.dll path.

I initially thought the SDK CLI mangled /mt into /m t, but that turned out to be the splat bug above — once fixed, /mt flows through dotnet msbuild unchanged. Verified by parsing the resulting binlog: 243 Csc invocations, no MSB1001, no error.

Parity proof

Built main@0f09213370 (Debug) end-to-end three times on Windows, with the bootstrapped stage 2 each time:

Snapshot	Command (after cd repo)
baseline_no_mt	full stage1+stage2: eng/cibuild_bootstrapped_msbuild.ps1 -msbuildEngine dotnet -onlyDocChanged $true -configuration Debug
with_mt	stage 2 only with /mt: same script, with -buildStage1 $false -stage2Properties "/mt"
baseline2_no_mt	stage 2 only without /mt: same script, with -buildStage1 $false

All three reuse the same stage-1 bootstrap, same source, same configuration, same ContinuousIntegrationBuild=true. The only varying input in with_mt is /mt. baseline2_no_mt is the control to measure pre-existing build non-determinism.

Hash-compared every file in artifacts/ (excluding log/, tmp/, obj/, SymStore/-staging, xsd/ cache, toolset/, TestResults/).

Result matrix (baseline_no_mt vs with_mt)

Subset	files	identical	differing	differing-but-noise
All non-log/tmp/obj files (whole artifacts/)	14,841	14,826	15	15
Shipping bin (bin\Microsoft.Build*, bin\MSBuild*, bin\StringTools, bin\MSBuild.Bootstrap, bin\MSBuildTaskHost)	1,628	1,628	0	0
Files inside the 4 shipping nupkgs (per file in the .nupkg ZIP, ignoring NuGet's psmdcp filename randomization)	71	71	0	0
Outer .nupkg ZIP raw SHA	4	0	4	4 (NuGet psmdcp GUID + relationship Id only)
SymStore/ (the symbol-server layout)	26	26	0	0
xsd/ (IntelliSense schemas published to VSSetup)	4	4	0	0

Repeating the same comparison between baseline_no_mt vs baseline2_no_mt (both -mp) yielded the exact same 15 differing files: 4 shipping nupkgs, 4 test CustomCheck nupkgs, 6 test DLLs, 1 .lnk shortcut. So none of the 15 diffs is /mt-induced — they reproduce between two -mp builds.

Per-shipping-nupkg breakdown

Each .nupkg was unzipped and compared file-by-file. After normalizing the random <32hex>.psmdcp filename and the random Id="R<16hex>" in _rels/.rels:

Package	files in .nupkg	identical (after normalisation)
Microsoft.Build.18.7.0-ci-26254-01.nupkg	15	15/15
Microsoft.Build.Framework.18.7.0-ci-26254-01.nupkg	22	22/22
Microsoft.Build.Templates.18.7.0-ci-26254-01.nupkg	13	13/13
Microsoft.NET.StringTools.18.7.0-ci-26254-01.nupkg	21	21/21

Covers every shipping .dll, .pdb, ref assembly, XML doc, nuspec, transitive build target, and .tlb we ship in the package family.

Compiler-invocation proof from the binlogs

Both builds emit artifacts/log/Debug/Build.binlog. I parsed each with the freshly-built Microsoft.Build BinaryLogReplayEventSource, captured every TaskCommandLineEventArgs, filtered to Csc/Vbc/Fsc, sorted by (project file, command line), and dumped to a text file:

$ Get-FileHash baseline_csc_invocations.txt mt_csc_invocations.txt -Algorithm SHA256

Algorithm Hash                                                             Path
--------- ----                                                             ----
SHA256    9387F541F8F53A65E8AE0DD0A1C69A77104902E30DF4045DC5764597F7622EE4 baseline_csc_invocations.txt
SHA256    9387F541F8F53A65E8AE0DD0A1C69A77104902E30DF4045DC5764597F7622EE4 mt_csc_invocations.txt

• 485 lines, 2,063,582 bytes each, identical SHA-256.
• Each file lists 242 Csc invocations with the full Roslyn command line (response file contents, references, sources, output paths, deterministic flags). Identical inputs to the C# compiler ⇒ identical outputs.

Macro stats from each binlog:

binlog property	baseline (no /mt)	with /mt
ProjectStarted events	1,382	1,382
Distinct project files built	48	48
Total task invocations	8,763	8,763
Csc+Vbc+Fsc invocations	242	242
MSBuild task invocations	535	535
Copy task invocations	243	243
WriteCodeFragment	232	232

What the 15 file diffs are (all pre-existing non-determinism, not /mt)

Verified by the baseline_no_mt vs baseline2_no_mt (-mp vs -mp) comparison:

1. NuGet psmdcp filename randomization in 4 shipping + 4 test nupkgs. NuGet writes package/services/metadata/core-properties/<random32hex>.psmdcp and a matching random Id in _rels/.rels. Same diff between two -mp builds. The .psmdcp content is identical; only its filename and the relationship Id differ.
2. Test DLL MVID + 2 bytes of PE TimeDateStamp in 6 test DLLs (Microsoft.Build.Engine.UnitTests, Microsoft.Build.Tasks.UnitTests, Microsoft.Build.CommandLine.UnitTests, each in net10.0 and net472). Test DLLs are not shipped. Verified with System.Reflection.Metadata.PEReader that PdbChecksum, CodeView GUID, embedded-PDB size, and every other byte of the assembly match — only the MVID and timestamp differ. Reproduces between two -mp builds.
3. VS with MSBuild.slnx.lnk Windows shortcut file regenerated each build.

Local verification of the script change

After all script changes, with no environment hacks:

> Remove-Item Env:DOTNET_HOST_PATH      # ensure unset outside the script
> .\eng\cibuild_bootstrapped_msbuild.ps1 -msbuildEngine dotnet -onlyDocChanged $true `
                                          -buildStage1 $false -configuration Debug `
                                          -skipTests -stage2Properties "/mt"

Result:

Build succeeded.
    213 Warning(s)
    0 Error(s)
Time Elapsed 00:02:13.04

(213 warnings are pre-existing analyzer warnings; same set/count appears in the no-/mt build.)

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[github-actions]

Correctness & Edge Cases — ISSUE

-skipTests incorrectly suppresses bootstrap creation (diverges from .sh reference)

The new PS1 code collapses $onlyDocChanged and $skipTests into one branch that always passes /p:CreateBootstrap=false:

if ($onlyDocChanged -or $skipTests) {
    & Build.ps1 -restore -build -ci /p:CreateBootstrap=false /nr:false ...
}

The shell-script reference (cibuild_bootstrapped_msbuild.sh) keeps them separate:

• onlyDocChanged=1 → passes /p:CreateBootstrap=false (bootstrap skipped)
• skipTests=true → does not pass /p:CreateBootstrap=false (bootstrap IS created, tests omitted)

Concrete failing call: .\cibuild_bootstrapped_msbuild.ps1 -skipTests
Observed: /p:CreateBootstrap=false is injected → bootstrap artifacts are not created.
Expected: Bootstrap is created, -test is simply omitted (matching .sh behaviour).

Any downstream step that consumes the bootstrap (including the /mt task-host scenario this PR specifically enables) will fail with a missing-bootstrap error.

Fix — separate the two cases to match the shell script:

if ($onlyDocChanged) {
    & $PSScriptRoot\Common\Build.ps1 -restore -build -ci /p:CreateBootstrap=false /nr:false `@properties` `@stage2Args`
}
elseif ($skipTests) {
    & $PSScriptRoot\Common\Build.ps1 -restore -build -ci /nr:false `@properties` `@stage2Args`
}
else {
    & $PSScriptRoot\Common\Build.ps1 -restore -build -test -ci /nr:false `@properties` `@stage2Args`
}

All other items reviewed (empty/whitespace $stage2Properties guard, [bool] -or short-circuit, env-var assignment) are correct.

Note

🔒 Integrity filter blocked 1 item

The following item were blocked because they don't meet the GitHub integrity level.

• #13678 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by Expert Code Review (on open) for issue #13678 · ● 5.7M

[JanProvaznik]

Refreshing PR (head should be e05c9c0: switch every PR CI bootstrapped job to -mt in stage 2 and remove the dedicated MT jobs).

addressed