[release/6.0] CompilerServer: fatal error - server reports different hash version than build task on linux-musl when using mono-backed runtime

[ayakael]

Description

Shared compilation with Roslyn fails on mono-backed runtime on linux-musl platforms. This only happens on .net6, thus suggesting it is simply a fix needing backporting.

Reproduction Steps

Within Alpine Linux linux-musl-x64 environment:

• Unpack bootstrap tar in ./bootstrap directory
• Attempt to build newtonsoft-json via:

./bootstrap/dotnet build Src/Newtonsoft.Json/Newtonsoft.Json.csproj /v:diag

• let it rip

Note that bootstrap tar was built using this aports script and #68424 to allow building with /p:PrimaryRuntimeFlavor=Mono

Expected behavior

Shared compilation should pass without issue

Actual behavior

Death

Regression?

Not a regression, per se. .net7 with mono has issues on linux-musl but this is not one of them.

Known Workarounds

Building with /p:UseSharedCompilation=false

Configuration

• .NET 6.0.109
• Alpine Linux Edge
• x64 and s390x have been tested to be broken
• Reproducible across multiple linux-musl platforms

Other information

No response

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[uweigand]

I've been seeing problems with the Roslyn compile server on .NET 6 because cross-process named mutexes are not implemented in the Mono runtime, leading to various race conditions with the compile server. See this issue: dotnet/roslyn#57002

The PR attached to that issue fixes the problem (by using a different synchronization mechanism), and has been added to Roslyn for .NET 7, but the .NET 6 version still needs this as a separate patch.

Not sure if that is exactly the same problem (the symptoms were a bit different), but might be worthwhile to try. (Note the patch needs to go into the Roslyn used by the host SDK as well as in the Roslyn that is being built via source-build.)

[ayakael]

I've been seeing problems with the Roslyn compile server on .NET 6 because cross-process named mutexes are not implemented in the Mono runtime, leading to various race conditions with the compile server. See this issue: dotnet/roslyn#57002

The PR attached to that issue fixes the problem (by using a different synchronization mechanism), and has been added to Roslyn for .NET 7, but the .NET 6 version still needs this as a separate patch.

Not sure if that is exactly the same problem (the symptoms were a bit different), but might be worthwhile to try. (Note the patch needs to go into the Roslyn used by the host SDK as well as in the Roslyn that is being built via source-build.)

The patch is added when crosscompiling using the prebuilt SDK from x64 to s390x. Is this enough, or does the SDK used for crosscompiling need it as well?

[uweigand]

The patch is added when crosscompiling using the prebuilt SDK from x64 to s390x.

Ah, I see. That's probably not it, then.

Is this enough, or does the SDK used for crosscompiling need it as well?

That's an Intel-hosted SDK? If so, it is presumably using CoreCLR, not Mono, and therefore does not need the patch.

[ayakael]

The patch is added when crosscompiling using the prebuilt SDK from x64 to s390x.

Ah, I see. That's probably not it, then.

Is this enough, or does the SDK used for crosscompiling need it as well?

That's an Intel-hosted SDK? If so, it is presumably using CoreCLR, not Mono, and therefore does not need the patch.

Right! I wonder if maybe installer isn't picking up the built roslyn packages? Is there a way I can confirm that the build sdk has the fixes?

[ayakael]

Following is used to build installer:

_installer() {
	msg "[$(date)] Building installer version $_installerver"
	cd "$builddir"/src/installer

	local args="
		-c Release
		/p:OSName=linux-musl
		/p:HostOSName=linux-musl
		/p:Architecture=$_dotnet_target
		/p:CoreSetupBlobRootUrl=file://$_downloaddir/
		/p:DotnetToolsetBlobRootUrl=file://$_downloaddir/
		/p:GitCommitHash=$(cat ./.git/HEAD)
		/p:GitCommitCount=$(grep GitCommitCount "$builddir"/git-info/installer.props | sed -E 's|</?GitCommitCount>||g' | tr -d ' ')
		/p:PublicBaseURL=file://$_downloaddir/
		/p:UseSharedCompilation=false
		"
	if [ "$_installerver" != "${_installerver##*-}" ]; then local args="$args /p:VersionSuffix=${_installerver##*-}"; fi
	if [ "$_dotnet_target" = "x86" ]; then local args="$args /p:DISABLE_CROSSGEN=True"; fi
	./build.sh $args
	mkdir  -p "$_downloaddir"/installer/$_installerver
	cp artifacts/packages/*/*/dotnet-sdk-$_pkgver_macro*.tar.gz "$_downloaddir"/installer/$_installerver
}

[uweigand]

The installer picks up the roslyn DLLs as part of the SDK tarball via the DotnetToolsetBlobRootUrl. That's created by the sdk package build, which in turn picks up the roslyn DLLs from NuGet - which is why you need to override NuGet.config in sdk to refer to the local directory where you've placed the packages generated from the roslyn build.

If any of that goes wrong, the build process can silently miss picking up the modified roslyn DLLs and just fall back to the versions on nuget.org ... To verify that the correct version was picked up, I'd simply compare the DLLs in your roslyn artifacts directory with the ones in the final SDK tarbal -- they should be identical.

[ayakael]

The installer picks up the roslyn DLLs as part of the SDK tarball via the DotnetToolsetBlobRootUrl. That's created by the sdk package build, which in turn picks up the roslyn DLLs from NuGet - which is why you need to override NuGet.config in sdk to refer to the local directory where you've placed the packages generated from the roslyn build.

If any of that goes wrong, the build process can silently miss picking up the modified roslyn DLLs and just fall back to the versions on nuget.org ... To verify that the correct version was picked up, I'd simply compare the DLLs in your roslyn artifacts directory with the ones in the final SDK tarbal -- they should be identical.

Awesome, this was it! Indeed, SDK wasn't picking the roslyn packages because of setting /p:ArcadeBuildFromSource=true. It was adding a nupkg cache line in NuGet.conf above local, thus overriding. Closing.