Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry Cloudformation StackSet Create when the service is busy #10969

Merged

Conversation

nijave
Copy link
Contributor

@nijave nijave commented Nov 21, 2019

Stack Sets only let a single stack per account be modified at the same
time. When an operation is in progress, an
OperationInProgressException is thrown that's similar to rate limiting
errors. This adds that exception type to the built-in Retryer in the AWS
SDK so it gets auto-retried like rate limits do

Community Note

  • Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
  • Please do not leave "+1" comments, they generate extra noise for pull request followers and do not help prioritize the request

Release note for CHANGELOG:

Automatically retry Cloudformation Stack Set instance operations when an operation is already in progress

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccXXX'

...

@nijave nijave requested a review from a team November 21, 2019 16:15
@ghost ghost added size/XS Managed by automation to categorize the size of a PR. needs-triage Waiting for first response or review from a maintainer. provider Pertains to the provider itself, rather than any interaction with AWS. labels Nov 21, 2019
@nijave nijave force-pushed the nv-retry-cloudformation-stackset-operations branch from 6fd0404 to 70789b2 Compare December 3, 2019 17:25
@abelmokadem
Copy link

Any updates on this PR? We are struggling without this feature.

@bflad bflad added the service/cloudformation Issues and PRs that pertain to the cloudformation service. label Jan 23, 2020
@nijave
Copy link
Contributor Author

nijave commented Jan 31, 2020

Any updates on this PR? We are struggling without this feature.

Haven't even heard anything back on an older PR with more votes yet, unfortunately

@mattsawyer77
Copy link

any idea when this might get merged? the current behavior's workaround of -parallelism=1 is very painful, at least for my use case of creating stackset instances across a large matrix of accounts and regions.

Base automatically changed from master to main January 23, 2021 00:56
@breathingdust breathingdust requested a review from a team as a code owner January 23, 2021 00:56
@ewbankkit ewbankkit added enhancement Requests to existing resources that expand the functionality or scope. and removed needs-triage Waiting for first response or review from a maintainer. labels Aug 27, 2021
@ewbankkit ewbankkit self-assigned this Aug 27, 2021
nijave and others added 2 commits August 27, 2021 09:43
Stack Sets only let a single stack per account be modified at the same
time. When an operation is in progress, an
`OperationInProgressException` is thrown that's similar to rate limiting
errors. This adds that exception type to the built-in Retryer in the AWS
SDK so it gets auto-retried like rate limits do
@ewbankkit ewbankkit force-pushed the nv-retry-cloudformation-stackset-operations branch from 70789b2 to 5327bab Compare August 27, 2021 13:51
@github-actions github-actions bot removed the service/cloudformation Issues and PRs that pertain to the cloudformation service. label Aug 27, 2021
@ewbankkit ewbankkit added the service/cloudformation Issues and PRs that pertain to the cloudformation service. label Aug 27, 2021
Copy link
Contributor

@ewbankkit ewbankkit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀.

% make testacc TEST=./aws TESTARGS='-run=TestAccAWSCloudFormationStackSetInstance_\|TestAccAWSCloudFormationStackSet_'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSCloudFormationStackSetInstance_\|TestAccAWSCloudFormationStackSet_ -timeout 180m
=== RUN   TestAccAWSCloudFormationStackSetInstance_basic
=== PAUSE TestAccAWSCloudFormationStackSetInstance_basic
=== RUN   TestAccAWSCloudFormationStackSetInstance_disappears
=== PAUSE TestAccAWSCloudFormationStackSetInstance_disappears
=== RUN   TestAccAWSCloudFormationStackSetInstance_disappears_StackSet
=== PAUSE TestAccAWSCloudFormationStackSetInstance_disappears_StackSet
=== RUN   TestAccAWSCloudFormationStackSetInstance_ParameterOverrides
=== PAUSE TestAccAWSCloudFormationStackSetInstance_ParameterOverrides
=== RUN   TestAccAWSCloudFormationStackSetInstance_RetainStack
=== PAUSE TestAccAWSCloudFormationStackSetInstance_RetainStack
=== RUN   TestAccAWSCloudFormationStackSet_basic
=== PAUSE TestAccAWSCloudFormationStackSet_basic
=== RUN   TestAccAWSCloudFormationStackSet_disappears
=== PAUSE TestAccAWSCloudFormationStackSet_disappears
=== RUN   TestAccAWSCloudFormationStackSet_AdministrationRoleArn
=== PAUSE TestAccAWSCloudFormationStackSet_AdministrationRoleArn
=== RUN   TestAccAWSCloudFormationStackSet_Description
=== PAUSE TestAccAWSCloudFormationStackSet_Description
=== RUN   TestAccAWSCloudFormationStackSet_ExecutionRoleName
=== PAUSE TestAccAWSCloudFormationStackSet_ExecutionRoleName
=== RUN   TestAccAWSCloudFormationStackSet_Name
=== PAUSE TestAccAWSCloudFormationStackSet_Name
=== RUN   TestAccAWSCloudFormationStackSet_Parameters
=== PAUSE TestAccAWSCloudFormationStackSet_Parameters
=== RUN   TestAccAWSCloudFormationStackSet_Parameters_Default
    provider_test.go:58: this resource does not currently ignore unconfigured CloudFormation template parameters with the Default property
--- SKIP: TestAccAWSCloudFormationStackSet_Parameters_Default (0.00s)
=== RUN   TestAccAWSCloudFormationStackSet_Parameters_NoEcho
    provider_test.go:58: this resource does not currently ignore CloudFormation template parameters with the NoEcho property
--- SKIP: TestAccAWSCloudFormationStackSet_Parameters_NoEcho (0.00s)
=== RUN   TestAccAWSCloudFormationStackSet_PermissionModel_ServiceManaged
    provider_test.go:58: API does not support enabling Organizations access (in particular, creating the Stack Sets IAM Service-Linked Role)
--- SKIP: TestAccAWSCloudFormationStackSet_PermissionModel_ServiceManaged (0.00s)
=== RUN   TestAccAWSCloudFormationStackSet_Tags
=== PAUSE TestAccAWSCloudFormationStackSet_Tags
=== RUN   TestAccAWSCloudFormationStackSet_TemplateBody
=== PAUSE TestAccAWSCloudFormationStackSet_TemplateBody
=== RUN   TestAccAWSCloudFormationStackSet_TemplateUrl
=== PAUSE TestAccAWSCloudFormationStackSet_TemplateUrl
=== CONT  TestAccAWSCloudFormationStackSetInstance_basic
=== CONT  TestAccAWSCloudFormationStackSet_Description
=== CONT  TestAccAWSCloudFormationStackSetInstance_RetainStack
=== CONT  TestAccAWSCloudFormationStackSetInstance_ParameterOverrides
=== CONT  TestAccAWSCloudFormationStackSetInstance_disappears_StackSet
=== CONT  TestAccAWSCloudFormationStackSetInstance_disappears
=== CONT  TestAccAWSCloudFormationStackSet_disappears
=== CONT  TestAccAWSCloudFormationStackSet_AdministrationRoleArn
=== CONT  TestAccAWSCloudFormationStackSet_basic
=== CONT  TestAccAWSCloudFormationStackSet_Tags
=== CONT  TestAccAWSCloudFormationStackSet_TemplateUrl
=== CONT  TestAccAWSCloudFormationStackSet_TemplateBody
=== CONT  TestAccAWSCloudFormationStackSet_Name
=== CONT  TestAccAWSCloudFormationStackSet_Parameters
=== CONT  TestAccAWSCloudFormationStackSet_ExecutionRoleName
--- PASS: TestAccAWSCloudFormationStackSet_disappears (53.86s)
--- PASS: TestAccAWSCloudFormationStackSet_basic (58.94s)
--- PASS: TestAccAWSCloudFormationStackSet_ExecutionRoleName (80.63s)
--- PASS: TestAccAWSCloudFormationStackSet_AdministrationRoleArn (84.43s)
--- PASS: TestAccAWSCloudFormationStackSet_Name (84.66s)
--- PASS: TestAccAWSCloudFormationStackSet_TemplateBody (85.11s)
--- PASS: TestAccAWSCloudFormationStackSet_Description (88.24s)
--- PASS: TestAccAWSCloudFormationStackSet_TemplateUrl (98.99s)
--- PASS: TestAccAWSCloudFormationStackSet_Parameters (126.25s)
--- PASS: TestAccAWSCloudFormationStackSetInstance_disappears_StackSet (126.55s)
--- PASS: TestAccAWSCloudFormationStackSet_Tags (132.50s)
--- PASS: TestAccAWSCloudFormationStackSetInstance_disappears (139.05s)
--- PASS: TestAccAWSCloudFormationStackSetInstance_basic (140.98s)
--- PASS: TestAccAWSCloudFormationStackSetInstance_RetainStack (203.91s)
--- PASS: TestAccAWSCloudFormationStackSetInstance_ParameterOverrides (261.14s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	268.802s

@ewbankkit
Copy link
Contributor

@nijave Thanks for the contribution 🎉 👏.

@ewbankkit ewbankkit merged commit 8166877 into hashicorp:main Aug 27, 2021
@github-actions github-actions bot added this to the v3.57.0 milestone Aug 27, 2021
@github-actions
Copy link

github-actions bot commented Sep 2, 2021

This functionality has been released in v3.57.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

@github-actions
Copy link

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jun 23, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement Requests to existing resources that expand the functionality or scope. provider Pertains to the provider itself, rather than any interaction with AWS. service/cloudformation Issues and PRs that pertain to the cloudformation service. size/XS Managed by automation to categorize the size of a PR.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants