Error configuring the backend "s3": RequestError: [...] TLS handshake timeout

[gcaraman-zz]

❌ [lambda_thub_user_create] Error configuring the backend "s3": RequestError: send request failed
caused by: Post https://sts.amazonaws.com/: net/http: TLS handshake timeout

Please update the configuration in your Terraform files to fix this error.
If you'd like to update the configuration interactively without storing

❌ [lambda_thub_user_create] the values in your configuration, run "terraform init".

❌ [lambda_thub_user_create] 

[eistrati]

@Leichtwind @ddimitrioglo @gcaraman I'd like to propose to address this issue by retrying failed terrahub command up to 3 times with exponential backoff approach. Thoughts?

[ddimitrioglo]

@eistrati generally this is on terraform's side, but we can re-run terraform init as much times as we want

[eistrati]

@ddimitrioglo that's correct, something is "fishy" between terraform and s3 ... and we can open an issue to terraform, but I fear that S3 is throttling multiple parallel terraform calls to S3 backend, so I'd like to propose an improvement by catching this issue and retrying in exponential backoff manner. Does this make sense?

[ddimitrioglo]

@eistrati it definitely does!

[gcaraman-zz]

Here is another sample error:

❌ [iam_thub_lambda_exec] caused by: read tcp 192.168.1.85:33808->54.231.120.27:443: read: connection reset by peer

❌ [ddb_vcs] SerializationError: failed to decode REST XML response

❌ [ddb_vcs] caused by: read tcp 192.168.1.85:33806->54.231.120.27:443: read: connection reset by peer

❌ Worker error: "`/home/grigore/.terrahub/terraform/0.11.7/terraform workspace list` failed with code 1"```

[mciuchitu]

Add option number of retries --retry (-r) for exponential backoff with default value 2.
Add default value in global config.

[gcaraman-zz]

Error: Error loading state: RequestError: send request failed
❌ [lambda_thub_mailchimp_subscribe] caused by: Get https://data-lake-terrahub-us-east-1.s3.amazonaws.com/terraform_workspaces/test/terraform/MitocGroup/terrahub-api/lambda_thub_mailchimp_subscribe/terraform.tfstate: EOF

[vcalmic]

Same error about S3 backend here hashicorp/terraform-provider-aws#4709

[eistrati]

One more use case:

❌ [lambda_github_repo_sync] 
Error: Error refreshing state: 1 error(s) occurred:

* data.external.lambda_github_repo_sync: 1 error(s) occurred:

* data.external.lambda_github_repo_sync: data.external.lambda_github_repo_sync: unexpected EOF

[eistrati]

❌ [s3_bucket] �[31mRequestError: send request failed

❌ [s3_bucket] caused by: Get https://data-lake-terrahub-us-east-1.s3.amazonaws.com/?prefix=terraform_workspaces%2F: net/http: TLS handshake timeout�[0m�[0m

❌ Worker error: "\u001b[31mRequestError: send request failed\n,caused by: Get https://data-lake-terrahub-us-east-1.s3.amazonaws.com/?prefix=terraform_workspaces%2F: net/http: TLS handshake timeout\u001b[0m\u001b[0m\n"```

[avozicov]

Reproduced again and this time there were network connection issue and terrahub did not try to re-run . In case this will be helpful I will attach log

[sqs_dbol_resource] This plan was saved to: /Users/alexandrvozicov/Projects/terrahub-api/.terrahub/sqs_dbol_resource/terraform.tfplan
[sqs_dbol_resource]
[sqs_dbol_resource] To perform exactly these actions, run the following command to apply:
[sqs_dbol_resource]     terraform apply "/Users/alexandrvozicov/Projects/terrahub-api/.terrahub/sqs_dbol_resource/terraform.tfplan"
[sqs_dbol_resource]
❌ [lambda_github_webhook_trigger] SerializationError: failed to decode REST XML response

❌ [lambda_github_webhook_trigger] caused by: read tcp 192.168.1.230:50508->52.216.165.75:443: read: connection reset by peer

❌ Worker error: "\u001b[31mSerializationError: failed to decode REST XML response\n,caused by: read tcp 192.168.1.230:50508->52.216.165.75:443: read: connection reset by peer\u001b[0m\u001b[0m\n"
Alexandrs-MacBook-Pro-2:Projects alexandrvozicov$

[avozicov]

This is fixed by implementing exponential backoff for init and plan actions.
I am marking as resolved but we will reopen this if will catch other use cases.

[eistrati]

💡 [lambda_pay_customer_retrieve] terraform apply -no-color -auto-approve=true [REDACTED]
[...]
❌ [lambda_pay_customer_retrieve] Failed to load backend: 
Error configuring the backend "s3": RequestError: send request failed
caused by: Post https://sts.amazonaws.com/: net/http: TLS handshake timeout```

[avozicov]

Fixed in and success for the following job and configured backend:
https://jenkins.mitocgroup.com/job/mitocgroup-terrahub-api/1886/consoleFull

[eistrati]

Checked.