state: Track module directory ModTime

[radeksimko]

Background

Currently, whenever we schedule ParseModuleConfiguration, ParseVariables, or ParseModuleManifest from anywhere, we just parse these files without checking whether we have already parsed the exact same file (with the same content) previously.

The scheduler already de-duplicates jobs, such that we never run the same job for the same folder more than once, if it's already in the queue. This is a good barrier, but we anticipate the following features to put more pressure on the scheduler and create more situations of duplicate work (same job for the same folder) scheduled more sparsely, which the scheduler itself has no chance of catching.

• Recognize inputs and outputs of uninitialized local modules #598
• Index uninitalized modules #724

To avoid wasting CPU on these unnecessary jobs we need some way of checking what files we have processed and whether they changed since.

The simplest way to do it would be reading, storing and comparing ModTime() (modification time) of the module folder.

Proposal

• Introduce DirModTime time.Time to state.Module
• state: Add lastModTime time.Time as a 2nd argument to ModuleStore.Add(modPath string)
• state: Introduce ModuleStore.HasChangedSince(modPath string, t time.Time) (bool, error)
• walker: Check ModTime() of every dir and pass it to HasChangedSince() before scheduling any jobs
• textDocument/didOpen handler: Check ModTime() of every dir and pass it to HasChangedSince() before scheduling any jobs

  terraform-ls/internal/langserver/handlers/did_open.go

  Lines 59 to 66 in 81b49a1

  	// We reparse because the file being opened may not match
  	// (originally parsed) content on the disk
  	// TODO: Do this only if we can verify the file differs?
  	modHandle := document.DirHandleFromPath(mod.Path)
  	jobIds, err := svc.indexer.DocumentChanged(modHandle)
  	if err != nil {
  	return err
  	}

• workspace/didChangeWorkspaceFolders handler: Check ModTime() of every dir and pass it to HasChangedSince() before scheduling any jobs

[radeksimko]

@jpogran raised some valid concerns/questions yesterday:

1. this may be difficult to implement on Windows as directory modification times are interpreted differently than on Unix
2. could we not just assume it's always up-to-date and fully rely on the LSP watch mechanism?

I want to try to find answer to (2) in particular as that looks like a solution that would better align with LSP spec, where the client watches for changes and notifies the server.

The only challenge is that we cannot easily treat pure existence of a module (as entry in memdb) as a guarantee that the module has all the data (i.e. that it was parsed, decoded etc.), so we'd likely need to do some deeper checking of the module - e.g. check the *State fields on the Module:

terraform-ls/internal/state/module.go

Lines 84 to 121 in ca335f5

	type Module struct {
	Path string
	ModManifest *datadir.ModuleManifest
	ModManifestErr error
	ModManifestState op.OpState
	TerraformVersion *version.Version
	TerraformVersionErr error
	TerraformVersionState op.OpState
	InstalledProviders InstalledProviders
	ProviderSchemaErr error
	ProviderSchemaState op.OpState
	RefTargets reference.Targets
	RefTargetsErr error
	RefTargetsState op.OpState
	RefOrigins reference.Origins
	RefOriginsErr error
	RefOriginsState op.OpState
	VarsRefOrigins reference.Origins
	VarsRefOriginsErr error
	VarsRefOriginsState op.OpState
	ParsedModuleFiles ast.ModFiles
	ParsedVarsFiles ast.VarsFiles
	ModuleParsingErr error
	VarsParsingErr error
	ModuleParsingState op.OpState
	VarsParsingState op.OpState
	Meta ModuleMetadata
	MetaErr error
	MetaState op.OpState

[radeksimko]

Closing in favour of #1002

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.