Implement canonical miscellaneous files project loader for non-file-based programs

[Copilot]

Resolves #80743
Resolves dotnet/vscode-csharp#7518
Resolves dotnet/vscode-csharp#7517

Implementation Plan for Canonical Miscellaneous Files Project Loader

Overview

Implementing issue #80743 to create a CanonicalMiscFilesProjectLoader that handles non-file-based-program miscellaneous files using a canonical project approach.

Completed ✅

• Understand current implementation and architecture
• Refactor LanguageServerProjectLoader to support custom primordial project handling
• Create CanonicalMiscFilesProjectLoader class
• Integrate CanonicalMiscFilesProjectLoader with FileBasedProgramsProjectSystem
• Build validation successful
• Code formatting applied
• Address all PR feedback:
  • Refactored canonical paths to single Lazy tuple
  • Updated loaded project verification logic
  • Changed method signatures to accept string paths
  • Removed host workspace references
  • Changed warnings to assertions
  • Made methods non-nullable
  • Simplified project XML
  • Added EnableFileBasedPrograms option check
  • Cleaned up comments
  • Removed unused ExecuteUnderGateAsync overload

Key Changes

1. LanguageServerProjectLoader.cs - Removed non-async ExecuteUnderGateAsync, cleaned comments
2. LanguageServerProjectSystem.cs - Removed "original behavior" comment
3. FileBasedProgramsProjectSystem.cs - Added option check, inlined variable, cleaned comments, updated method calls
4. CanonicalMiscFilesProjectLoader.cs - Complete refactor per feedback
5. LoadedProject.cs - Added GetProjectSystemProject() accessor

Implementation Highlights

Improvements from feedback:

• Lazy initialization of canonical project paths ensures thread-safe single initialization
• Always using misc workspace (never host) for canonical files
• Assertions instead of warnings for conditions that should never happen
• Non-nullable return types for better type safety
• Simplified XML template (basic class library without custom targets)
• Gated behind EnableFileBasedPrograms option for safe rollout

Original prompt

I'd like you to implement @dotnet/roslyn/issues/80743 - but there are a few points to be careful of

1. The loading and design time build of the misc file needs to not block the AddMiscDocument call to avoid blocking the LSP queue on doing a design time build.
2. The AddMiscDocument call must return a document. If nothing is loaded, it should return something similar to the current primordial document in the FBP loader.
3. We should re-use as much as we can from the LanguageServerProjectLoader

Generally the idea is to generate an empty .cs document in temp and use that as the canonical project for misc file project loading.

For the implementation, I would recommend the following approach.
We can add a CanonicalMiscFilesProjectLoader that implements LanguageServerProjectLoader. Generally this type is responsible for loading the canonical project and adding misc files to it. It needs to handle adding misc files before the canonical project is loaded (and transfer them to the canonical project when done).

When a misc document is added, we need to check if the canonical project has been loaded (e.g. the canonical project path exists in _loadedProjects with a non primordial state and some targets). If it is loaded, we should apply a change to the workspace to add a new miscellaneous document with the provided text to the project and return that document instance.

If the project does not exist at all, we need to create a primordial project and start creating the canonical project. The primordial project should function very similarly to the FileBasedProgramsProjectSystem version, but use the generated temp cs document as the initial misc document, and add the requested misc file to that project as well. The canonical project load should call BeginLoadingProjectWithPrimordialAsync with the generated temp cs document as well. The requested misc document should be returned (not the generated temp cs doc)

If the project exists, but in the primordial state (meaning a different misc file was already added and triggered generation), we should instead just add the new document to the primordial project and return that.

This should all be done under the _gate in LanguageServerProjectLoader (may need to expose a new method to allow taking an action on the loaded project state to the subclasses).

That ensures that the primordial project has the correct files, but we still need to ensure misc files added to the primordial project transfer to the canonical project once loaded. To do that we refactor how the LanguageServerProjectLoader currently removes the primordial project when the real one is loaded. Currently it just deletes the primordial project (see https://github.com/dotnet/roslyn/blob/main/src/LanguageServer/Microsoft.CodeAnalysis.LanguageServer/HostWorkspace/LanguageServerProjectLoader.cs#L324). Instead we should make this abstract. The FBP provider would use the current implementation, but the new CanonicalMiscFilesProjectLoader would take all the misc documents we added in the primordial project (anything that doesn't match the canonical doc path) and add them to the host project, then delete the primordial project.

For the implementation of TryLoadProjectInMSBuildHostAsync, it should be similar to the one in FileBasedProgramsProjectSystem, but use a static csproj xml for a class library template instead of the _projectXmlProvider.

Removing a misc document should also be done under the base _gate and generally should attempt to remove the file from the workspace (which should handle it either being in the primordial or canonical project). The canonical project should never be removed and stays around for the lifetime of the process.

FileBasedProgramsProjectSystem will hold on to and (lazily) create an instance of the CanonicalMiscFilesProjectLoader.

In FileBasedProgramsProjectSystem.AddMiscellaneousDocumentAsync if we have a c# URI that is virtual (not a file on disk - check the ParsedUri) or is not VirtualProjectXmlProvider.IsFileBasedProgram(documentPath, textAndVersion.Text), we will call into the canonical loader's add misc document method instead of the FBP one.

In FileBasedProgramsProjectSystem.IsMiscellaneousFilesDocumentAsync it should also check the CanonicalMiscFilesProjectLoader to see if it is loaded there.

In FileBasedProgramsProjectSystem.TryRemoveMiscellaneousDocumentAsync it should also delegate to the CanonicalMiscFilesProjectLoader to see if it can be unloaded there.

Concurrency notes:

1. the calls to check if a file is misc, add a misc doc, and remove a misc document are never called in parallel. However they can run concurrently with code in the LanguageServerProjectLoader that runs the design time build and loads the project. Hence it being important that modifications to the loaded project run under the _gate there.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[RikkiGibson]

I find this somewhat suspect, it doesn't seem like one project system should "contain" another project system.

[dibarbet]

Another potential path is to create multiple misc files provider. But that is a bit of a larger refactoring as we have to update the rest of the queue and providers to be chained so to speak. Can look into it if you feel strongly about this.

[RikkiGibson]

I will try to take another dive into this area soon (perhaps checking out the change, stepping through some things, observing changes in state over time, etc.), and follow up if I have any ideas to make things simpler and more robust.

[RikkiGibson]

This is the last bit I am not quite satisfied with. It's possible it is fine, I just need to put a little more scrutiny on it. I was wondering, is the Lazy partially functioning to break a cycle during initialization?

[dibarbet]

No - it can very well not be lazy now actually. Artifact of when the ctor actually did something. I will make it not lazy, but not sure that covers everything you were thinking about.

[dibarbet]

I think generally the right path here is to allow multiple misc files providers. The LSPWorkspaceManager would be responsible for iterating them to see which one could handle the document (likely ordered, with FBP first, then canonical).

Then the FBP provider here doesn't need to know about the canonical provider.

It's a little larger change though and I'd prefer to do in a followup if you are OK with that.

[RikkiGibson]

The state management of all the different ways a file can be misc over time, is pretty hard, and severely under-tested. It will be great to be able to put more attention into that area.

I think just getting unstuck on bringing in more test coverage, how to represent more complex scenarios in the test harness, etc, will do us an immense amount of good.

[RikkiGibson]

What occurs to me is: we don't have a need to continually update the LoadedProject for the canonical misc file and such. We don't need to do more DTBs of it over time and so on.

Therefore I suspect that what we should actually do, is: keep a single field in FBPProjectSystem of type ProjectLoadState, which essentially holds the DTB state of our canonical misc project. Once it transitions to Loaded, we basically never touch it any more. We update the workspace only based on files being added/removed and that is fine and good. We can extract the bits from the base which allow us to respond to the DTB completion and transition the state of the field.

Basically I think there are so many steps that should not happen the same way for the canonical, as they do for ordinary projects or FBPs, that we might just want to write a completely new code path which simply doesn't include all the stuff we don't need to do.

• Everything related to detecting an existing LoadedProject in memory and updating it.
• Everything around unloading.
• Everything around handling non-sdk-style projects. (BuildHostKindThatWeWantedButDidNotGet etc)
• Everything about telemetry (I think we don't want telemetry events for loading the canonical).
• Special handling of restore. Let's avoid a popup for "Restoring Canonical.cs", or at least use a special message in that case.
  • It's actually not clear to me if/when we want to delete the canonical project. Why not just leave it there for the next session? I realize this is probably contradicting prior comments I made.

Observations from trying to run the language server from the PR branch (some of this is repetition from the previous section)

• We get a popup for "Restoring Canonical.cs". Maybe this should probably be special cased to not pop anything up.
• I tried defining class C1 in one misc file and using it in another, it didn't work. This seems surprising.
• Saving an untitled file caused a restore to kick off which did not terminate.
• It also caused the an error popup to appear for the following log output

2025-10-31 14:45:47.354 [error] [textDocument/diagnostic] [LSP] Failed to get language for untitled:Untitled-1 with language 
2025-10-31 14:45:47.355 [debug] [textDocument/diagnostic] [LSP] Found 0 diagnostics for DocumentDiagnosticSource: c:\Users\rikki\src-backup\fbp-scratch\untitled.cs in Miscellaneous Files
2025-10-31 14:45:47.355 [debug] [textDocument/diagnostic] [LSP] Microsoft.CodeAnalysis.LanguageServer.Handler.Diagnostics.Public.PublicDocumentPullDiagnosticsHandler finished getting diagnostics
2025-10-31 14:45:47.355 [debug] [textDocument/diagnostic] [LSP] Request handler completed successfully.
2025-10-31 14:45:47.355 [debug] [textDocument/diagnostic] [LSP] Found 0 diagnostics for DocumentDiagnosticSource: c:\Users\rikki\src-backup\fbp-scratch\untitled.cs in Miscellaneous Files
2025-10-31 14:45:47.355 [debug] [textDocument/diagnostic] [LSP] Found 0 diagnostics for DocumentDiagnosticSource: c:\Users\rikki\src-backup\fbp-scratch\untitled.cs in Miscellaneous Files
2025-10-31 14:45:47.355 [debug] [textDocument/diagnostic] [LSP] Microsoft.CodeAnalysis.LanguageServer.Handler.Diagnostics.Public.PublicDocumentPullDiagnosticsHandler finished getting diagnostics
2025-10-31 14:45:47.356 [debug] [textDocument/diagnostic] [LSP] Microsoft.CodeAnalysis.LanguageServer.Handler.Diagnostics.Public.PublicDocumentPullDiagnosticsHandler finished getting diagnostics
2025-10-31 14:45:47.357 [debug] [textDocument/diagnostic] [LSP] Request handler completed successfully.
2025-10-31 14:45:47.357 [debug] [textDocument/diagnostic] [LSP] Request handler completed successfully.
2025-10-31 14:45:47.357 [error] [textDocument/diagnostic] [LSP] System.Exception: Failed to get language for textDocument/diagnostic
2025-10-31 14:45:47.358 [error] Request textDocument/diagnostic failed.
  Message: Failed to get language for textDocument/diagnostic
  Code: -32000 
[object Object]
2025-10-31 14:45:47.358 [error] Document pull failed for text document untitled:Untitled-1
  Message: Failed to get language for textDocument/diagnostic
  Code: -32000 

[dibarbet]

Basically I think there are so many steps that should not happen the same way for the canonical, as they do for ordinary projects or FBPs, that we might just want to write a completely new code path which simply doesn't include all the stuff we don't need to do

It's definitely possible to do, but we would end up keeping a decent amount of the logic from the abstract base project system (e.g. transitioning the project). imho its kind of a wash either way, the amount of code we can reuse from the base proj system is similarish to the amount that doesn't apply.

Let's avoid a popup for "Restoring Canonical.cs"

I can definitely get behind this.

It's actually not clear to me if/when we want to delete the canonical project. Why not just leave it there for the next session? I realize this is probably contradicting prior comments I made.

I don't think we should use it for multiple sessions. What happens if you have two different instances of VSCode trying to access the canonical project? It may work fine, but you seems likely to cause file locking headaches and other issues. Think its completely reasonable to have each instance use its own and clean it up when done.

I tried defining class C1 in one misc file and using it in another, it didn't work. This seems surprising.

This seems to work for me:

Saving an untitled file caused a restore to kick off which did not terminate.

Not seeing this - a restore of the canonical project?

It also caused the an error popup to appear for the following log output

Unrelated and existing issue dotnet/vscode-csharp#8675

[RikkiGibson]

• Everything related to detecting an existing LoadedProject in memory and updating it.

There is a need to reload after the restore completes. So I don't think it would cut away as much as I was speculating.

I don't think we should use it for multiple sessions. What happens if you have two different instances of VSCode trying to access the canonical project? It may work fine, but you seems likely to cause file locking headaches and other issues. Think its completely reasonable to have each instance use its own and clean it up when done.

I buy what you are saying here. I think the tooling should be able to tolerate multiple instances being open, especially something like VS and VS Code having the same solution open at the same time. But I get that it tends to cause problems.

re: the endless restore, and the misc files not seeing each other--I will try to repro and see what might have been going on with that.

[dibarbet]

Let's avoid a popup for "Restoring Canonical.cs"

I can definitely get behind this.

This requires some additional changes. The popup is managed client side and needs client side changes. However I have a plan to move the popup management entirely server-side (using work done progress), which would allow us to suppress it only on the server.

Would prefer to handle that in a separate PR. RIght now the popup for the canonical project should only show up once per session, so isn't too bad of an experience.

[RikkiGibson]

Change LGTM. Looking forward to more goodness in follow ups.

[dibarbet]

merging so I can work on followups

[jcouv]

@dibarbet In the future, consider trimming the commit message before merging. It doesn't seem very useful to have all this in the git history: 8060956