azurerm_data_factory_linked_service_data_lake_storage_gen2 use_managed_identity false doesn't setup a working connection

[jwshive]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Affected Resource(s)

• `Terraform v0.15.1
  on darwin_amd64

• provider registry.terraform.io/hashicorp/azurerm v2.57.0`

Terraform Configuration Files

resource "azurerm_storage_account" "working-storage_account" {
  name                     = "${local.agency-name}${local.application-name}${local.environment}wksa"
  resource_group_name      = "client-rg"
  location                 = local.resource-location
  account_tier             = "Standard"
  account_replication_type = "RAGRS"
  account_kind             = "StorageV2"
  is_hns_enabled           = "true"
  access_tier              = "Hot"
  allow_blob_public_access = false
}

resource "azurerm_data_factory_linked_service_data_lake_storage_gen2" "LS-storage-account" {
  name                 = "LS_${azurerm_storage_account.working-storage_account.name}"
  resource_group_name  = "client-rg"
  data_factory_name    = azurerm_data_factory.primary-data_factory.name
  use_managed_identity = false
  tenant               = data.azurerm_client_config.current_client_config.tenant_id
  url                  = "https://${azurerm_storage_account.working-storage_account.primary_dfs_host}"
}

Debug Output

Panic Output

Expected Behaviour

I expect I should be able to test the connection and it connects successfully to adlsgen2

Actual Behaviour

Connection fails with

ADLS Gen2 operation failed for: The data factory client-dev-df has no MI provisioned. Please update the data factory following: https://docs.microsoft.com/en-us/azure/data-factory/data-factory-service-identity#generate-service-identity-using-powershell. Account: 'clientdevwksa'..
The data factory client-dev-df has no MI provisioned. Please update the data factory following: https://docs.microsoft.com/en-us/azure/data-factory/data-factory-service-identity#generate-service-identity-using-powershell
The managed identity of entity client-dev-df was not found.
Activity ID: df301b98-79d4-4d3c-8a56-ae3b56c9fe73.

Steps to Reproduce

terraform destroy on the linked service
terraform apply

Important Factoids

On the linked service settings pane, the host is the correct url and the account key is populated.
If I change the account selection method from manual to 'from azure subscription' and select the same storage account, I see it briefly say loading key and then I can test and get a successful connection.
When I go back into the settings pane, the url is exactly the same and the account key is populated, but I cannot see what the actual account key is.

I'm assuming it's getting the wrong account key from somewhere, but I'm not sure where that would be or how I can provide it the correct key.

When I do the above and get a successful connection, I am able to run a terraform plan again on the unaltered code and there are no changes detected.

References

• #0000

[njuCZ]

@jwshive accoridng to doc: https://docs.microsoft.com/en-us/azure/data-factory/connector-azure-data-lake-storage#managed-identity If you want to use managed Identity authentication, you should assign the relevant roles. For example:

resource "azurerm_role_assignment" "test" {
  scope                = azurerm_storage_account.test.id
  role_definition_name = "Storage Blob Data Owner"
  principal_id         = azurerm_data_factory.test.identity.0.principal_id
}

Then I believe you could get the successfully connection.

If you want to use the account key authentication, PR #12136 has add support for it. You could upgrade to the latest version and have a try!

[jwshive]

I don't know that I can use the latest version. We run all of our code through azure devops and I can't get it to init on any version above 0.14.11.

That said, I still don't feel like this is working properly. To "fix" my issue I just have to toggle it off and back on again. So the code doesn't seem to be the problem.

[njuCZ]

@jwshive What I mean is there are different ways to do it:

1. you could upgrade the latest azurerm provider and use storage account key authentication
2. If you want to use managed identity authentication, according to the doc, you should assign a role. I have tested it successfully
3. use service principal then assign the role

[tombuildsstuff]

Closing this out since this appears to be resolved?

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.