AWS: Certificate not found even though it was just created

[nathanielks]

Related: #2490

I deleted the certificate using AWS's cli. The output shows it was created but then when it goes to create the ELB, AWS says that the certificate doesn't exist.

Here are the logs.

[phinze]

Thanks for the report - we've seen this issue in several different places. We'll have to see if there's something we can do after the certificate create to ensure it's ready to be attached to an ELB.

[nathanielks]

👍

[mitchellh]

I'm playing with this now. The AWS logs say that it can take up to 15 minutes for certificates to propagate, which would indeed be a really sad thing. I'm trying to find a way for us to detect this since the API doesn't seem to expose it.

[rgabo]

Just ran into the same issue. awscli is quite misleading because aws iam list-server-certificates immediately lists the certificate that was just created, even though when referenced in other services (ELB in our case), it will throw an error. Thanks for looking into this, Mitchell.

[xuwang]

I have the same issue, one workaroud is to wait for a while in local provisioner :

resource "aws_iam_server_certificate" "mycert" {
  name = "mycert"
  certificate_body = "${file("mycert.pem")}"
  private_key = "${file("mycert-key.pem")}"

  provisioner "local-exec" {
    command = <<EOF
        echo # Sleep 10 secends so that mycert is propagated by aws iam service
        echo # See https://github.com/hashicorp/terraform/issues/2499 (terraform ~v0.6.1)
        sleep 10
EOF
  }
}

10 seconds wait time seems to work for me. May a terraform function can be introduced to handle the need for external waiting? e.g. depend_on with a waiting time.

[farridav]

This is affecting me too, and I also got he same output from aws iam list-server-certificates ..Just posted on these 2 related issues #3275 and #3412 .. I will try the temporary sleep fix.. but keen to hear if there is a proper solution I can use 👍

[catsby]

I've submitted #3898 as a patch for #3275 and #3412. Unfortunately it is largely a bandaid. IAM resources are notoriously eventually consistent for our experiences; API calls return "success" but follow up calls that reference the ARN return 404s :/

[catsby]

#3898 was merged, so I'm going to close this.
Let me know if you're still seeing this on the master branch, otherwise the fix will rollout in the next release (should be next week)

[nathanielks]

👍

[nathanielks]

@catsby Finally had the opportunity to test #3898. Having upgraded to 0.6.8, I'm unfortunately still seeing the error. Adding a 10 second sleep to the certificate "resolves" the issue though ¯_(ツ)_/¯

[catsby]

Adding a 10 second sleep to the certificate "resolves" the issue though

Can you elaborate on how you're doing this? The retry logic in #3898 should retry the creation for up to 1 minute, so, I'm not sure where a sleep for 10 seconds is helping, that a retry for 1 minute wouldn't also fix. Do you see the warning "[WARN] Error creating ELB Listener with SSL Cert, retrying" in the logs?

Or is this for deletion? This issue has crossed some wires 😄

[nathanielks]

Sure can, @catsby! Here's the whole resources for funzies:

resource "aws_iam_server_certificate" "domains" {                                                                                                               
    count = "${var.load_balancer_count}"                                                                                                                        

    name = "${var.environment}-${element(split(" ",var.certificate_names), count.index)}"                                                                       

    certificate_body = "${file(format("%s/%s/%s.%s", var.certificates_base, var.environment, element(split(" ",var.certificate_names), count.index), "crt"))}"  
    certificate_chain = "${file(format("%s/%s/%s.%s", var.certificates_base, var.environment, element(split(" ",var.certificate_names), count.index), "chain"))}
    private_key = "${file(format("%s/%s/%s.%s", var.certificates_base, var.environment, element(split(" ",var.certificate_names), count.index), "key"))}"       

    provisioner "local-exec" {                                                                                                                                  
        command = "sleep 10"                                                                                                                                    
    }                                                                                                                                                           
}                                                                                                                                                               

It's on creation. I'm not sure I'll have time to re-run everything, but I'll do that as soon as I can and get back with you on the ELB Listener warning!

[catsby]

@nathanielks I attempted to reproduce with this:

• https://gist.github.com/catsby/cd0d296c572e65a4a7f0

but I can't. Let me know what I'm missing

[jurajseffer]

I think I'm hitting this in terraform v0.10.8 when trying to assign a newly saved IAM server certificate to an ALB listener. Terraform gives me an ARN for aws_iam_server_certificate but it fails to apply the change to listeners with an AWS error saying the certificate with given ARN cannot be found. When I manually look at the certificates available via AWS console, I can see it being there and subsequent plan and apply works fine so it looks like a problem with eventual consistency.

[cloudvant]

Yeah, I'm hitting this in terraform v0.10.8 as well. It works on macOS and Ubuntu 16, but fails on Centos/Jenkins running on EC2.

[mattchilds1]

It seems I'm experiencing the same thing as @jurajseffer . The only different thing I'm doing is pulling the cert in from another AWS account with a separate provider.

Terraform returns the correct ARN, pass's it into the ELB Listener but then fails with the certificate not found error.

@catsby

EDIT: Apologies, I've now seen that AWS don't support using of certificates across trusted accounts. A change to get a different error message could still be good though?

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.