Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IAM Role creation race stil present in spot instance requests #3554

Closed
ghost opened this issue Feb 28, 2018 · 10 comments
Closed

IAM Role creation race stil present in spot instance requests #3554

ghost opened this issue Feb 28, 2018 · 10 comments
Labels
bug Addresses a defect in current functionality. service/ec2 Issues and PRs that pertain to the ec2 service.
Milestone

Comments

@ghost
Copy link

ghost commented Feb 28, 2018

This issue was originally opened by @stadeltobi as hashicorp/terraform#17457. It was migrated here as a result of the provider split. The original body of the issue is below.


Terraform Version

Terraform v0.10.2

resource "aws_spot_instance_request" "quick-sir" {
  ami = "ami-5652ce39"
  instance_type = "t2.xlarge"
  spot_price = ".15"
  wait_for_fulfillment = true
  vpc_security_group_ids = <...>
  key_name = <...>
  iam_instance_profile = "${aws_iam_instance_profile.foo.id}"
  user_data = "${data.template_file.cloud-init.rendered}"
}

...

Additional Context

I'm having the equivalent issue that is described in hashicorp/terraform#15341 when, instead of trying to create an aws_instance, I use a aws_spot_instance_request.

The same error ("Invalid Parameter Value") is thrown, so I'm assuming that the race condition described above was fixed for regular (on-demand) instances still prevails for spot_instance_requests.

Also, the workaround suggested by @spanktar makes the problem go away.

The problem goes away if I switch the spot request resource with an aws_instance resource, using the exact same parameters (except for the ones like spot_price that don't apply for aws_instance

References

@radeksimko radeksimko added bug Addresses a defect in current functionality. service/iam Issues and PRs that pertain to the iam service. labels Feb 28, 2018
@bflad bflad added service/ec2 Issues and PRs that pertain to the ec2 service. and removed service/iam Issues and PRs that pertain to the iam service. labels Feb 28, 2018
@bflad
Copy link
Contributor

bflad commented Feb 28, 2018

@stadeltobi can you confirm which version of the AWS provider you are running? It does look like the aws_spot_instance_request resource of current provider (v1.10.0) only has 15 seconds of retry where we normally retry for a minute or two when dealing with IAM eventual consistency.

@stadeltobi
Copy link
Contributor

stadeltobi commented Feb 28, 2018

@bflad thanks for getting back to me!

It's version 1.10.0 of the aws provider.

And you're probably right: in my setup, the error is thrown exactly 15 seconds after attempting to create the respective aws_spot_instance_request ...

@bflad
Copy link
Contributor

bflad commented Feb 28, 2018

Nice - let's bump that up to a minute to match other retry logic 😄 Would you feel comfortable submitting a PR? No acceptance test changes required.

https://github.com/terraform-providers/terraform-provider-aws/blob/fefe3470fe9aadca6e84a79bcc56b46a3f420952/aws/resource_aws_spot_instance_request.go#L139

@stadeltobi
Copy link
Contributor

stadeltobi commented Feb 28, 2018

sure, why not :)

#3561

@bflad bflad closed this as completed in 74ae053 Feb 28, 2018
bflad added a commit that referenced this issue Feb 28, 2018
increase timeout in spot instance creation to 1 min - fix #3554
@bflad bflad added this to the v1.11.0 milestone Feb 28, 2018
@bflad
Copy link
Contributor

bflad commented Feb 28, 2018

PR above has been merged and will release in v1.11.0 of the AWS provider, likely releasing next week. Of course you're more than welcome to try out a custom local build too. 😉

@stadeltobi
Copy link
Contributor

I'll do that; thanks a lot for the quick response and the help.

@stadeltobi
Copy link
Contributor

So now I did (built c56c678, so a few commits after the PR was merged).

Works like a charm.

Thanks again 😄

@bflad
Copy link
Contributor

bflad commented Mar 9, 2018

This has been released in version 1.11.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

@asgoel
Copy link

asgoel commented Dec 10, 2018

Hi all, we recently started seeing this happening again, on provider version 1.42. Given that we haven't updated the provider recently, could this be due to some slowness on AWS' side?

@ghost
Copy link
Author

ghost commented Apr 1, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Apr 1, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Addresses a defect in current functionality. service/ec2 Issues and PRs that pertain to the ec2 service.
Projects
None yet
Development

No branches or pull requests

4 participants