preinstall terraform providers into tfcache to avoid concurrent writes at runtime

[kapilt]

else xdist concurrency can result in text busy file error while one worker is downloading and another is concurrently executing the provider.

[kapilt]

trying to handle this automatically is tricky, we do lock files around create, destroy when running w/ xdist, but thats around a given terraform fixture name, but a given fixture maps 1 to m on providers, and we don't know apriori all the providers.

brainstorming options

• we could move all the inits to the main thread when running xdist, I'm not sure if the controller node does collection though, and that would be required.
• parse the terraform on all fixtures, to find all the providers, and then utilize that, but still requires controller node collection.
• disable tf plugin cache dir, so we avoid the centralized writes on init, we currently always set this value.
• ???

[kapilt]

it doesn't appear controller node does any collection, so an alternate thought is we just try a initialization lock in a worker, and run through init on all the fixtures. but we'll need to synchronize with the other workers, possibly via the lock timeout.

[kapilt]

a bit more research, I think we just need to drop the shared plugin cache dir when using xdist.

hashicorp/terraform#31964
hashicorp/terraform#32901

[kapilt]

given the behavior I'm seeing in 1.5, where one terraform init's provider download doesn't suffice for another root module, we probably just need to nuke it entirely. this does mean lots of extraneous downloads :/ but we should ask terraform to support this use case better directly.