Removing Microsoft.CodeAnalysis.ExternalAccess.RazorCompiler

[jaredpar]

This external access DLL was added to allow experimentation between compiler and the razor source generator. The external access approach was chosen because it was a proven method in other parts of the Roslyn code base and seemed like a good fit for the compiler.

After our experiences in 17.8 around ABI changes though we've discovered this is not a good fit for the compiler. The compiler does not own it's deployment, instead it's deployed / hosted by a number of applications: VS, VS Code, Workspaces, dotnet format, etc ... All of these hosts now must be aware of and ship a copy of the MS.CA.EA.RazorCompiler DLL. This DLL is not part of the core NuGet package set which means the hosts must find the correct version and keep it up to date with the compiler (version mismatches can fail due to IVT differences). This leads to a very fragile deployment model and it bit us many times during the 17.8 release cycle.

Going forward the compiler will likely use reflection or possibly [Experimental] for future experiments. In the short term though our desire is to remove this DLL from our deployments. To do this we're going to take the following measures:

17.9

• Publish the MS.CA.EA.RazorCompiler NuPkg to nuget.org so existing hosts have a place to download the 4.8.0 compatible version.
• Take the APIs that exist in MS.CA.EA.RazorCompiler today and move them into a new type in MS.CA under the namespace Microsoft.CodeAnalysis.Experimental. These APIs will be internal.
• Change the MS.CA.EA.RazorCompiler assembly to point their implementation to those methods. At this point the MS.CA.EA.RazorCompiler assembly is just a shell that redirects method calls into MS.CA.
• Change the razor source generator to use reflection to access the APIs inside of MS.CA.EA. If those APIs do not exist then fall back to calling MS.CA.EA.RazorCompiler. This fallback is necessary for situations where the razor source generator from 17.9 is run on the 17.8 compiler.

17.10

• Delete MS.CA.EA.RazorCompiler entirely from razor. From here on out it will use reflection to grab the experimental API.
• Delete the MS.CA.EA.RazorCompiler project from roslyn

[jjonescz]

• At this point the MS.CA.EA.RazorCompiler assembly is just a shell that redirects method calls into MS.CA.

It's not that simple, because apart from methods, there's also struct HostProductionContext which is part of the API. I guess it can be moved to Experimental and type-forwarded from EA. It seems it would be an error if razor was then recompiled (and didn't yet use reflection) because the HostProductionContext.AddOutput method wouldn't be accessible from razor; but it shouldn't binary-break razor I hope (will verify locally).

[jjonescz]

Hm, doesn't seem to be working. I tried building an app with razor with host outputs enabled + roslyn with the change (using our toolset packages) and I'm getting:

CSC : warning CS8784: Generator 'RazorSourceGenerator' failed to initialize. It will not contribute to the output and compilation errors may occur as a result. Exception was of type 'MethodAccessException' with message 'Attempt by method 'Microsoft.NET.Sdk.Razor.SourceGenerators.RazorSourceGenerator.Initialize(Microsoft.CodeAnalysis.IncrementalGeneratorInitializationContext)' to access method 'System.Action`3,System.Threading.CancellationToken>..ctor(System.Object, IntPtr)' failed.'.

That might be also because of the toolset packages not working here correctly. Don't know how else to easily verify this, though.

Anyway, if it's a real error, then we would probably need to leave that type in the EA assembly. But then Experimental would need to reference EA, that won't work.

Also - host outputs are currently disabled in razor since the rename (I have a PR to re-enable them here: dotnet/razor#9447) - maybe we can just leave them disabled and remove the EA layer entirely in 4.9.0?

[jjonescz]

Razor doesn't use the host outputs since 17.9P1 (since this PR: dotnet/razor#9129). Hence, the new plan is to completely remove the EA reference from razor in 17.9, leave it in roslyn just-in-case in 17.9 and remove it from roslyn too in 17.10. If we want to use host outputs again, we will use reflection.

[jaredpar]

Now that the razor side is merged moving this issue to 17.10. That is when we can complete the second part of the issue which is removing the DLL entirely from this repo.