Since provider version 1.34.0 terraform plan is really slow
#5822
Comments
rmldsky
changed the title
terraform plan is really slow after upgrading providerterraform plan is really slow
|
I have upgraded terraform to |
|
I've checked on a different machine, with fresh install of both I'll try to check if there is anything I can do on the El Capitan machine to fix it. Any clues why it might be an issue on that platform? |
|
Have you tried running Terraform with debugging enabled? Maybe the debug log will offer some clues. Please feel free to Gist it and post them here. If you are worried about any sensitive data, it can be encrypted with the HashiCorp GPG Key or redacted as necessary. |
bflad
added
waiting-response
|
FYI, I am exactly in the same situation. Terraform 0.11.8 and provider.aws 1.35 is really slow. Any plan or apply action takes 10 times longer than before. |
|
@bflad ah, did not though about this (duh!), will do tomorrow. @supasteev0 are you also on macOS El Capitan? |
|
@rmldsky No, I'm on Linux Fedora 25. |
|
@bflad I did run Sample occurrence looks something like below and usually lasts 20sec: What is strange(?) is that those lines with |
|
Admittedly, its a little hard to discern what is occurring with only the provided snippet of logs, but hopefully we can narrow this down! 😅
The interleaved output is occurring since by default Terraform runs with a parallelism of 10. Maybe tuning the I would also check in the debug logs to see if the AWS Go SDK is returning |
|
@bflad just FYI not to leave you hanging, I'll get back on this issue, this week. |
|
@rmldsky any update? |
|
@bflad sorry, took longer then expected for me to get back on it. Please see this encrypted (keybase/hashicorp) log output: Let me know if you can access this file and decrypt it. Also I did it with
vs If you'd like me to run it with some other flags please let me know. |
|
@rmldsky hmm something isn't quite right with that file or maybe I'm doing something wrong? GPG is only decrypting the first 7077 lines and reporting some errors: |
|
Got it! So its interesting that there seems to be 20 seconds of delay between every request and the response, e.g. Version 1.34.0 of the AWS provider was the first release that started using Go 1.11 (instead of Go 1.10) for builds, so maybe something changed in connection handling? Is there anything potentially strange about your networking environment (dual IPv4+IPv6 enabled, proxies, etc)? |
|
@bflad no, nothing special. I also tried this on the same machine on different network and it was same story. Also not really sure what to look for in the network setup. Same network and machine on I'll try to find someone in the office to test it out on different OS. |
|
These upstream Go bug reports seem interesting:
Can you verify if your local DNS configuration has multiple name servers or if IPv6 is enabled locally? |
|
Great catch. I had few DNS entries and left only Also I've check on a friend's machine (1.37.0, macOS Mojave BUT with go runtime 1.10.?) and it was as fast as |
|
The upstream Go issues mentioned above seem to indicate it was a Go 1.11 regression with DNS handling so it should be machine specific based on DNS configuration. The machine's Go runtime shouldn't matter unless you are manually building the provider and running that custom built provider. It seems Go 1.11.1 is slated to include a fix, so we'll keep this issue open to track the officially released version of the AWS provider that is built via that updated Go version. |
bflad
added
upstream
|
Hello guys. I also had multiple DNS entries (thank you wifi networks ...). Anyway, thank you for the feedback, problem solved on my side, but might be worth looking into this Go version issue indeed. |
|
Go 1.11.1 has been released, which should include this fix. I'll switch our build processes to use the updated Go version and hopefully this will be part of the version 1.39.0 release. |
|
The DNS issue starting in version 1.34.0 (built with Go 1.11.0) should now be resolved in version 1.39.0 (built with Go 1.11.1), released just now. 🎉 |
|
This has been released in version 1.39.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example: provider "aws" {
version = "~> 1.39.0"
}
# ... other configuration ... |
|
I confirm version 1.39.0 has solved the issue on my side. Terraform is now working flawlessly, even with multiple DNS entries in /etc/resolv.conf. |
|
This does not seem fixed for me. With multiple DNS entries, here's what I'm seeing on a dirt simple Terraform example: provider "aws" {
region = "eu-west-1"
}
data "aws_region" "current" {}
data "template_file" "foo" {
template = "foo"
}
output "foo" {
value = "${data.template_file.foo.rendered}"
}
output "region" {
value = "${data.aws_region.current.name}"
}Time: And the exact same code with one DNS entry: Terraform version: |
|
@brikis98 was it a case for you with previous (< |
|
I had this same problem. My issue was one of the entries in my DNS config wasn't resolving. Emptying it out to be solely |
|
@nathanielks response fixed my probleme |
|
Locking this issue to encourage fresh bug reports, if necessary. 👍 As noted above, please double check your DNS resolvers! |
Community Note
Terraform Version
0.11.8Problem
Not sure if it's me or something under the hood. I was running
0.11.8/1.27.0and upgraded provider to1.35.0. Immediate after upgrade I ranterraform planand it took very long time to finish. I blamed network at first, but after some time I ran it again with same result. I've downgraded plugin back to1.27.0and it is fast again ... Not sure if it is isolated issue or something was introduced in one of the versions following1.27.0:/I run it on the same subset of resources and state plan (remote S3). Also not sure if it's network-slow or locally-slow
1.27.0time terraform plan1.35.0time terraform planI'll try to upgrade version-by-version and see when it starts happening for me.
The text was updated successfully, but these errors were encountered: