Azure keyvault gets created but secret creation doesnt wait for availability, and crashes

[pixelicous]

Terraform Version

Terraform v0.11.1
AzureRM version 1.0

Terraform Configuration Files

resource "azurerm_key_vault" "test_keyvault" {
    provider = "azurerm.operations"
    name = "test-${var.MyNumber}-kv"
    location = "North Europe"
    resource_group_name = "${azurerm_resource_group.opstenant_resource_group.name}"

    sku {
        name = "standard"
    }

    tenant_id = "${var.tenant_id}"

  access_policy {
    tenant_id = "${var.tenant_id}"
    object_id = "${var.group_devops_id}"

    key_permissions = [
      "get",
    ]

    secret_permissions = [
      "get","set","list","delete"
    ]
  }

  access_policy {
    tenant_id = "${var.tenant_id}"
    object_id = "${var.spn_devops_id}"

    key_permissions = [
      "get",
    ]

    secret_permissions = [
      "get","set","list","delete"
    ]
  }

    access_policy {
    tenant_id = "${var.tenant_id}"
    object_id = "${var.tfs_prod_spn}"

    key_permissions = [
      "get",
    ]

    secret_permissions = [
      "get","set","list","delete"
    ]
  }


  enabled_for_deployment = true
  enabled_for_template_deployment = true

tags = "${var.resource_tags}"
}

################### SECRETS

resource "azurerm_key_vault_secret" "tenant_admuser_secret" {
  provider = "azurerm.operations"
  name      = "${var.admUser}"
  value     = "${var.admPass}"
  vault_uri = "${azurerm_key_vault.test_keyvault.vault_uri}"
  depends_on = ["azurerm_key_vault.test_keyvault"]

  tags = "${var.resource_tags}"
}

Crash Output

1 error(s) occurred:

• module.MyModule.module.TestModule.azurerm_key_vault.test_keyvault: 1 error(s) occurred:
• azurerm_key_vault.ops_keyvault: dial tcp: lookup test-00006-kv.vault.azure.net: no such host

Expected Behavior

AzureRM keyvault should be created and secret should wait for DNS propogation

Actual Behavior

I receive the crash.
If i re run the same command, it works, because now the keyvault DNS hostname is replicated across azure dns servers

Steps to Reproduce

Create .tf files with keyvault and secret
Terraform apply

Important Factoids

The files are fine, a second terraform apply rerun creates the secrets successfully.
The "depends_on" part was added to test maybe it will help waiting for keyvault availbility
This issue started out of the blue, like a new terraform version issue.

[tombuildsstuff]

hey @pixelicous

Thanks for opening this issue

The files are fine, a second terraform apply rerun creates the secrets successfully.

Out of interest - how long are you waiting between creating the Key Vault via Terraform and re-running? Also do you have a corporate proxy involved caching DNS queries at all?

As part of creating the Key Vault we poll to ensure the DNS is available for up to 2 minutes (technically we're checking that the TCP Socket is open on port 443 and then returning so the service should be available). My guess here would be that the DNS hasn't finished replicating fully and so perhaps we need to wait for a given number of DNS queries to be successful (e.g. wait for 10 DNS queries to be returned) before assuming it's successful.

This issue started out of the blue, like a new terraform version issue.

Would it be possible to know if this is still an issue with the previous version? (and which previous version this worked with?) You can achieve this by specifying the version in the Provider block e.g.:

provider "azurerm" {
  version = "1.0.0"
}

The "depends_on" part was added to test maybe it will help waiting for keyvault availbility

Given there's an Implicit Dependency by referencing the Key Vault directly (e.g. vault_uri = "${azurerm_key_vault.test_keyvault.vault_uri}") - the depends_on (an Explicit Dependency) shouldn't be needed. In either case the Dependency will exist and the Key Vault Secret is waiting for the Key Vault to exist before creating - so the logic is wrong in the Key Vault resource.

In order to work out how best to proceed here, it'd be good to confirm that this is a case of the DNS not being fully replicated (for instance, if there's a corporate proxy involved we may need to add additional delays to ensure that's handled) :)

Thanks!

[pixelicous]

@tombuildsstuff hey mate, first thanks for the reply and the details.

1. I didn't really use a timer between executions, but longer than 2minutes i reckon.
2. Not sure about proxy, i think not, we do however have a firewall.. I will try to do it with a different version of azurerm provider.
   It is a very weird issue though, what i do not understand is how come there are locations where it is always working from, and other which it never does. From my PC, and my colleague's, it is working fine always.. but we have a build server with the same powershell/terraform version from which it always fails.. i also spun up a VM in azure (which is routed through our firewall) but that also fails.. When we re-apply terraform on the azure vm or build server, it will work.. so it is some sort of dns replication issue..
3. The depends_on clause was added just as a test, i understand it is implicitly depending on the keyvault.

[pixelicous]

@tombuildsstuff i just checked, the VM which is in Azure is using the Azure provided DNS, we also go directly to the internet from it, no proxies along the way

[pixelicous]

@tombuildsstuff I just had this issue on my PC as well. On my PC out of around 20-30executions non failed up till today, i had an execution which failed with the same error..

[pixelicous]

@tombuildsstuff there is no reply for around a month now.. but im referencing the original issue at azure's side Azure/azure-powershell#1524

This can be resolved by querying the relevant object and returning success only after it is properly created

[tombuildsstuff]

hey @pixelicous

Just to let you know that v1.4.0 of the Azure Provider has been released which includes the fix for this :)

Thanks!

[pixelicous]

@tombuildsstuff thanks! kudos for quick releases between 1.3.3 and 1.4 🔥
I just tested creation of secrets and certificates and it works great as far as i can see! 😃

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!