8.0 VMR all legs failing in runtime with detected package downgrade

[Winniexu01]

Internal/8.0.1xx build: https://dev.azure.com/dnceng/internal/_build/results?buildId=2433696&view=results

/vmr/src/runtime/artifacts/source-build/self/src/src/tasks/WasmAppBuilder/WasmAppBuilder.csproj : error NU1605: Warning As Error: Detected package downgrade: System.Collections.Immutable from 8.0.0 to 7.0.0. Reference the package directly from the project to select a different version.  [/vmr/src/runtime/artifacts/source-build/self/src/Build.proj]
/vmr/src/runtime/artifacts/source-build/self/src/src/tasks/WasmAppBuilder/WasmAppBuilder.csproj : error NU1605:  WasmAppBuilder -> Microsoft.Build.Tasks.Core 17.8.5 -> System.Collections.Immutable (>= 8.0.0)  [/vmr/src/runtime/artifacts/source-build/self/src/Build.proj]
/vmr/src/runtime/artifacts/source-build/self/src/src/tasks/WasmAppBuilder/WasmAppBuilder.csproj : error NU1605:  WasmAppBuilder -> System.Collections.Immutable (>= 7.0.0) [/vmr/src/runtime/artifacts/source-build/self/src/Build.proj]

/vmr/src/runtime/artifacts/source-build/self/src/src/tasks/Microsoft.NET.Sdk.WebAssembly.Pack.Tasks/Microsoft.NET.Sdk.WebAssembly.Pack.Tasks.csproj : error NU1605: Warning As Error: Detected package downgrade: Microsoft.Build.Framework from 17.8.5 to 17.8.3. Reference the package directly from the project to select a different version.  [/vmr/src/runtime/artifacts/source-build/self/src/Build.proj]
/vmr/src/runtime/artifacts/source-build/self/src/src/tasks/Microsoft.NET.Sdk.WebAssembly.Pack.Tasks/Microsoft.NET.Sdk.WebAssembly.Pack.Tasks.csproj : error NU1605:  Microsoft.NET.Sdk.WebAssembly.Pack.Tasks -> Microsoft.Build 17.8.5 -> Microsoft.Build.Framework (>= 17.8.5)  [/vmr/src/runtime/artifacts/source-build/self/src/Build.proj]
/vmr/src/runtime/artifacts/source-build/self/src/src/tasks/Microsoft.NET.Sdk.WebAssembly.Pack.Tasks/Microsoft.NET.Sdk.WebAssembly.Pack.Tasks.csproj : error NU1605:  Microsoft.NET.Sdk.WebAssembly.Pack.Tasks -> Microsoft.Build.Framework (>= 17.8.3) [/vmr/src/runtime/artifacts/source-build/self/src/Build.proj]

/vmr/src/runtime/artifacts/source-build/self/src/src/tasks/WasmAppBuilder/WasmAppBuilder.csproj : error NU1605: Warning As Error: Detected package downgrade: System.Collections.Immutable from 8.0.0 to 7.0.0. Reference the package directly from the project to select a different version.  [/vmr/src/runtime/artifacts/source-build/self/src/Build.proj]
/vmr/src/runtime/artifacts/source-build/self/src/src/tasks/WasmAppBuilder/WasmAppBuilder.csproj : error NU1605:  WasmAppBuilder -> Microsoft.Build 17.8.5 -> System.Collections.Immutable (>= 8.0.0)  [/vmr/src/runtime/artifacts/source-build/self/src/Build.proj]
/vmr/src/runtime/artifacts/source-build/self/src/src/tasks/WasmAppBuilder/WasmAppBuilder.csproj : error NU1605:  WasmAppBuilder -> System.Collections.Immutable (>= 7.0.0) [/vmr/src/runtime/artifacts/source-build/self/src/Build.proj]

/vmr/src/runtime/artifacts/source-build/self/src/src/tasks/WasmAppBuilder/WasmAppBuilder.csproj : error NU1605: Warning As Error: Detected package downgrade: System.Reflection.Metadata from 8.0.0 to 7.0.0. Reference the package directly from the project to select a different version.  [/vmr/src/runtime/artifacts/source-build/self/src/Build.proj]
/vmr/src/runtime/artifacts/source-build/self/src/src/tasks/WasmAppBuilder/WasmAppBuilder.csproj : error NU1605:  WasmAppBuilder -> Microsoft.Build 17.8.5 -> System.Reflection.Metadata (>= 8.0.0)  [/vmr/src/runtime/artifacts/source-build/self/src/Build.proj]
/vmr/src/runtime/artifacts/source-build/self/src/src/tasks/WasmAppBuilder/WasmAppBuilder.csproj : error NU1605:  WasmAppBuilder -> System.Reflection.Metadata (>= 7.0.0) [/vmr/src/runtime/artifacts/source-build/self/src/Build.proj]

/vmr/src/runtime/artifacts/source-build/self/src/src/tasks/AotCompilerTask/MonoAOTCompiler.csproj : error NU1605: Warning As Error: Detected package downgrade: System.Reflection.Metadata from 8.0.0 to 7.0.0. Reference the package directly from the project to select a different version.  [/vmr/src/runtime/artifacts/source-build/self/src/Build.proj]
/vmr/src/runtime/artifacts/source-build/self/src/src/tasks/AotCompilerTask/MonoAOTCompiler.csproj : error NU1605:  MonoAOTCompiler -> Microsoft.Build 17.8.5 -> System.Reflection.Metadata (>= 8.0.0)  [/vmr/src/runtime/artifacts/source-build/self/src/Build.proj]
/vmr/src/runtime/artifacts/source-build/self/src/src/tasks/AotCompilerTask/MonoAOTCompiler.csproj : error NU1605:  MonoAOTCompiler -> System.Reflection.Metadata (>= 7.0.0) [/vmr/src/runtime/artifacts/source-build/self/src/Build.proj]

The related SDK files are not generated due to the errors mentioned above, CentOSStream8_Online_CurrentSourceBuiltSdk_x64 and Fedora38_Offline_CurrentSourceBuiltSdk_x64 are failed:
Copy Previous Build step:

found 0 files
Finishing: Copy Previous Build

Prep the Build step:

tar (child): /mnt/vss/_work/1/vmr/prereqs/packages/archive/dotnet-sdk-*.tar.gz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now

Build step:

Custom SDK '/vmr/.dotnet/dotnet' does not exist or is not executable

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[MichaelSimons]

@NikolaMilosavljevic - Can you jump on this? TIA. This is related to dotnet/runtime#100595

[NikolaMilosavljevic]

It did not repro when I built locally with latest changes: https://dev.azure.com/dnceng/internal/_git/dotnet-dotnet/commit/75831229196c2d6d803830ad1558d20e673a16be?refName=refs/heads/release/8.0.1xx

I'm not sure why that build had a different commit that is now missing.

[MichaelSimons]

Look at the internal/release/8.0.1xx branch - https://dev.azure.com/dnceng/internal/_git/dotnet-dotnet?path=/&version=GBinternal/release/8.0.1xx&_a=history

[NikolaMilosavljevic]

It seems that runtime now gets PVP flow for Microsoft.Build which bumps it to live (n-1) version (17.8.5) which brings 8.0.0 dependencies of System.Reflection.Metadata and System.Collections.Immutable.

17.8.5 was resolved from previously-source-built source.

Runtime defines S.C.I and S.R.M dependencies version as 7.0.0. https://github.com/dotnet/runtime/blob/ca4f0fe37455882baa00c75b1ef30a7ff1494457/eng/Versions.props#L127

    <SystemCollectionsImmutableVersion>7.0.0</SystemCollectionsImmutableVersion>

Same for the other assembly.

[lewing]

These are toolset dependencies, the Tasks in question go into workloads and are run in-process with the sdk/VS durring application build

in net9 this is explicitly called out by the properties that are used
https://github.com/dotnet/runtime/blob/main/eng/Versions.props#L135-L139

cc @akoeplinger

[NikolaMilosavljevic]

Updating runtime, to remove direct package references (S.C.I and S.R.M.) was not enough. MSbuild repo was regressed with the same runtime PR (dotnet/runtime#100595), and downgraded the versions from 17.8.5 to 17.8.3. Runtime PR added msbuild intermediate package reference with SourceBuild tag. That caused VMR sync to pick the old MSBuild SHA, instead of the one specified in sdk repo.

I tried, locally, to create a patch for runtime to remove the intermediate dependency, to check if vmr-sync would do the right thing. It did not update msbuild to a newer SHA. This runtime change is still needed and I will include it in the same patch with other runtime change mentioned above. However, we need to find a way to update VMR. If sync is not going to bring the correct msbuild SHA, can we update it directly?

@MichaelSimons @premun