Speed up by caching plugins and providers

[atlantisbot]

Issue by @lkysow
Monday Sep 04, 2017 at 01:47 GMT
Migrated from hootsuite/atlantis#141
Why was it migrated?

---

With the latest terraform >= 0.9 providers come as plugins that need to be downloaded. Right now, we're using terraform init which downloads the plugins into the current directory under .terraform/plugins/{os}_{runtime} for every project.

If we move the downloaded plugin to ~/.terraform.d/plugins/ (or ~/.terraform.d/plugins/{os}_{runtime} then terraform won't download the plugin again for any project.

So my proposed solution is:

• on every terraform init, check if there are any files in {pwd}/.terraform/plugins/os_runtime other than lock.json. If so, move all of them to ~/.terraform.d/plugins/.

Now tf will check ~/.terraform.d/plugins/ before downloading any plugins so it will be back up to speed. Since tf will download any plugins that aren't already existing, we can rely on it to do the downloading.

[atlantisbot]

Comment by @anubhavmishra
Tuesday Sep 05, 2017 at 21:13 GMT

---

Some providers are versioned and people might request two different versions of the same provider. Do we account for that? Just a question.

[atlantisbot]

Comment by @lkysow
Tuesday Sep 05, 2017 at 22:26 GMT

---

Yeah terraform names the plugin files depending on the version so we can have multiple files cached and tf will just use whatever version it needs

[atlantisbot]

Comment by @automaticgiant
Tuesday Dec 05, 2017 at 01:43 GMT

---

can we not just use https://www.terraform.io/docs/configuration/providers.html#provider-plugin-cache ?
well, in theory.
i was trying to set TF_PLUGIN_CACHE_DIR and terraform init to cache deps in the dockerfile, but the env doesn't persist to child tf. sad. we have a corporate proxy, so this is awkward. so now i'm loading them into ~/.terraform.d/plugins.

[atlantisbot]

Comment by @lkysow
Tuesday Dec 05, 2017 at 05:37 GMT

---

Nice! I didn't know that was possible. I think it probably makes sense to always execute terraform with the TF_PLUGIN_CACHE_DIR variable set to "$HOME/.terraform.d/plugin-cache"

[dheerajbhaskar]

Can this be made the default? Why does this basic feature have to be configured on every server, dev machine, and docker container where terraform is used?

PS: Sorry, if this came across as a rant, I'm likely in a bad mood. I tried rephrasing the sentence to be as neutral as possible.

[lkysow]

This is the default in Atlantis.

[wosiu]

It's been 4 years, so: is it still a default?
I don't understand how terraform native plugin caching can be used in atlantis, taking into account this is not safe for concurrent operations like atlantis does.
References:
hashicorp/terraform#31964

[wosiu]

Yup, it seems this is still enabled by default unfortunately, which causes probably all of these issues:
#2412
#4866
#3536
#3547