walker: Index uninitialized modules

[radeksimko]

Fixes #724

Depends on hashicorp/terraform-schema#126

---

The PR changes the walker such that it indexes any directory it finds with >0 *.tf files in it. This has a few side effects I had to account for:

• as a result of indexing many more modules, this uncovered a race condition where reference collection would run before metadata decoding, resulting in no references being collected.
  • Reference collection jobs were moved under Defer to provide that guarantee.
• Some E2E tests assume that the existence of .terraform dir without other files would trigger indexing, but the new behaviour is more correct in not indexing such a directory. The reason we start with empty directory with just .terraform is because it means we can more easily provide the actual test configuration as part of the test via didOpen and we don't have to deal with copying/creating test configs in FS.
  • The same function responsible for "bootstrapping" a directory to become indexable was updated to also create empty.tf
  • We could update all these E2E tests such that they don't rely on walker at all - i.e. we don't do any indexing on initialize, but that would also mean we have to find a different way of loading the schema, e.g. by inserting it into memdb as part of each test.
  • It is questionable whether these tests are in fact truly E2E when they mock out the Terraform CLI calls. I think the best solution might be having two versions of most tests, where (1) is truly E2E and installs and executes real Terraform CLI, and (2) mocks out enough memdb state such that tests can run offline
  • We so far used many E2E tests as a way to test the walker, so we'd have to also add some unit tests to the walker if we no longer run it as part of the E2E tests.

---

Indexing Before

Indexing After

---

I filed a few follow-up issues to deal with the potential performance hit resulting from indexing more modules:

• state: Track module directory ModTime #989
• terraform/module: Introduce new job to parse provider versions from lock file #992
• decoder: Pick core schema only based on required_version constraint #993

[dbanck]

Nice work! This is really fast & fun to use.

I tried different numbers of modules and couldn't see any performance impact

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.