Replace internal watcher with LSP's DidChangeWatchedFilesRegistrationOptions

[radeksimko]

Background

Currently the LS implements its own watcher to catch any changes in what are typically non-user-editable, but important metadata (under .terraform/). Knowing when this metadata changes is important so that the server can re-load installed modules or provider schemas from there.

This became less "critical" feature in 0.16.0 after #454 where we started pre-building schemas of ~200 official+partner providers. However, there is still some amount of users who rely on schemas sourced locally.

The LSP says the following about server-side file watching:

Servers are allowed to run their own file system watching mechanism and not rely on clients to provide file system events. However this is not recommended due to the following reasons:

• to our experience getting file system watching on disk right is challenging, especially if it needs to be supported across multiple OSes.
• file system watching is not for free especially if the implementation uses some sort of polling and keeps a file system tree in memory to compare time stamps (as for example some node modules do)
• a client usually starts more than one server. If every server runs its own file system watching it can become a CPU or memory problem.
• in general there are more server than client implementations. So this problem is better solved on the client side.

Previously LSP however didn't provide any way for server to tell the client what files it wants to receive changes for. This generally makes sense for editable files (such as *.tf or *.tfvars), but it doesn't work for the files described above. i.e. we shouldn't expect clients to have list of file patterns additional to *.tf and *.tfvars.

Since LSP 3.15 there's a way to do this:

• client sends dynamicRegistration: true in DidChangeWatchedFilesClientCapabilities as part of initialize request
• server responds on initialize request
• server dynamically registers DidChangeWatchedFilesRegistrationOptions by sending client/registerCapability with appropriate globPattern per

  terraform-ls/internal/terraform/datadir/paths.go

  Lines 10 to 21 in f49cd99

  	var pluginLockFilePathElements = [][]string{
  	// Terraform >= 0.14
  	{".terraform.lock.hcl"},
  	// Terraform >= v0.13
  	{DataDirName, "plugins", "selections.json"},
  	// Terraform >= v0.12
  	{DataDirName, "plugins", runtime.GOOS + "_" + runtime.GOARCH, "lock.json"},
  	}
  	var manifestPathElements = []string{
  	DataDirName, "modules", "modules.json",
  	}

  to the client
• server processes workspace/didChangeWatchedFiles notifications

Proposal

• internal/service: add support for didChangeWatchedFiles #790 / Support workspace/didChangeWatchedFiles #15
• Implement dynamic capability registration either for all capabilities or just for watched file capability
• Remove builtin watcher
• Update architecture docs

Related

hashicorp/vscode-terraform#405

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.