AKS service principal creation not handling async

[jjcollinge]

When running the AKS example for the first time I got the following error:

error: Plan apply failed: containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="ServicePrincipalNotFound" Message="Service principal clientID: ****-****-****-**** not found in Active Directory tenant ****-****-****-****, Please see https://aka.ms/acs-sp-help for more details."

When I re-ran pulumi up it worked. I believe this is due to the asynchronous nature of AAD service principal creation and propagation rather than a bug in the dependency graph. AAD is a particularly problematic resource and I believe has an SLA of ~2 mins for entity availability. It'd be worth having some validate and retry logic such as that in the Azure CLI (Azure/azure-cli#1499) before returning from the create method.

[lukehoban]

Looking into this. It sounds like it may be an issue in the underlying terraform-provider-azurerm with some wait logic missing from the resource creation path.

[lukehoban]

Root cause is https://github.com/terraform-providers/terraform-provider-azurerm/issues/1635.

[jjcollinge]

@lukehoban thanks, looks like you're right. I see the latest Pulumi azure resources are based on TF 1.13 so they should include the fix TF released in 1.12. Do you think their fix didn't actually resolve the underlying issue or am I missing something?

[lukehoban]

Do you think their fix didn't actually resolve the underlying issue or am I missing something?

I don't believe there is a fix for terraform-providers/terraform-provider-azurerm#1635 yet. I believe the fix for hashicorp/terraform-provider-azurerm#1647 addressed a similar case but for a different API (role assignment, not AKS).

[lukehoban]

I'm going to close this out - since it is really tracking terraform-providers/terraform-provider-azurerm#1635 in upstream getting fixed/merged and then adopting that into Pulumi. We expect to be adopting latest providers into Pulumi on a continuous basis in the near term, so will just want to push on the upstream issue to get fixed.

@jjcollinge Feel free to reopen if you'd like to have an issue to track on the Pulumi side - else tracking this in upstream is probably sufficient.

[lukehoban]

Actually - appears this was just fixed a couple days ago - and will be in the next release: hashicorp/terraform-provider-azurerm#2204

[lukehoban]

My comment above about this being fixed is incorrect - https://github.com/terraform-providers/terraform-provider-azurerm/issues/1635 is still open. That said, I believe there is nothing Pulumi-specific we can do here.

The "best current workaround" described there is to sleep for 30 seconds. That can be done in Pulumi with:

let adApp = new azuread.Application("aks");
let adSp = new azuread.ServicePrincipal("aksSp", { applicationId: adApp.applicationId });
let adSpPassword = new azuread.ServicePrincipalPassword("aksSpPassword", {
    servicePrincipalId: adSp.id,
    // ...
});
let delayedAdSpPasswordValue = adSpPassword.value.apply(async (val) => {
    // Wait for 30s
    console.log("Waiting for 30s for AD Service Principal eventual consistency...");
    await new Promise(resolve => setTimeout(resolve, 30000));
    return val;
});
let aks = new azure.containerservice.KubernetesCluster("aksCluster", {
    //...
    servicePrincipal: {
        // ...
        clientSecret: delayedAdSpPasswordValue,
    },
});

[mastoj]

@lukehoban, I guess applies to ManagedCluster in azure-native as well? Or is it a fix for it now that I have missed?