backend/s3: Configure AWS Client MaxRetries and provide enhanced S3 NoSuchBucket error message #20338
Conversation
…oSuchBucket error message The AWS Go SDK automatically provides a default request retryer with exponential backoff that is invoked via setting `MaxRetries` or leaving it `nil` will default to 3. The terraform-aws-provider `config.Client()` sets `MaxRetries` to 0 unless explicitly configured above 0. Previously, we were not overriding this behavior by setting the configuration and therefore not invoking the default request retryer. The default retryer already handles HTTP error codes above 500, including S3's InternalError response, so the extraneous handling can be removed. This will also start automatically retrying many additional cases, such as temporary networking issues or other retryable AWS service responses. Changes: * s3/backend: Add `max_retries` argument * s3/backend: Enhance S3 NoSuchBucket error to include additional information
|
We've generally been considering v0.11 functionality to be frozen as we work towards the v0.12.0 release, with an exception for the new "remote" backend used by Terraform Enterprise as part of the free state storage rollout, so I'm a little reluctant to merge this for that reason. However, since it appears that this change reduces the risk of a failed state write, and is thus a safety improvement for S3 backend users, I think we could make an exception if we're confident that this will not cause any visible change in behavior for those who were not experiencing this problem. I don't have enough context to make that call, but if you do then I'm happy to have this merged into the 0.11 branch, though note that 0.11 releases are currently being driven entirely by the free state storage feature rollout (since the Terraform Core team is 100% focused on 0.12.0 release preparation) and so we will need to wait until there's a release driven by that work before this fix would become available to v0.11 users. |
…ider-aws to github.com/hashicorp/aws-sdk-go-base References: * #20374 * #20338 * #19992 Backports the S3 backend code from master to v0.11 relating to the AWS Go SDK, which removes the circular dependency with the Terraform AWS Provider and adds retry logic to the backend to prevent temporary networking or AWS service issues from failing unnecessarily. Updated via code change then: ``` go mod tidy go mod vendor ```
|
Superseded by #20659 👍 |
…ider-aws to github.com/hashicorp/aws-sdk-go-base References: * #20374 * #20338 * #19992 Backports the S3 backend code from master to v0.11 relating to the AWS Go SDK, which removes the circular dependency with the Terraform AWS Provider and adds retry logic to the backend to prevent temporary networking or AWS service issues from failing unnecessarily. Updated via code change then: ``` go mod tidy go mod vendor ```
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further. |
ghost
locked and limited conversation to collaborators
Closes #19992
This is a backport of commit (ed37d07) which was merged into master via #19951 and was to resolve a fairly common issue as noted there.
The AWS Go SDK automatically provides a default request retryer with exponential backoff that is invoked via setting
MaxRetriesor leaving itnilwill default to 3. The terraform-aws-providerconfig.Client()setsMaxRetriesto 0 unless explicitly configured above 0. Previously, we were not overriding this behavior by setting the configuration and therefore not invoking the default request retryer.The default retryer already handles HTTP error codes above 500, including S3's InternalError response, so the extraneous handling can be removed. This will also start automatically retrying many additional cases, such as temporary networking issues or other retryable AWS service responses.
Changes:
max_retriesargument