Create inventory of service principals and direct files access in Azure for a Spark session

[dipankarkush-db]

In Azure, data access is authorized using service principals using Spark session settings.

Spark session level settings
spark.conf.set("fs.azure.account.auth.type","OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type","org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id","")
spark.conf.set("fs.azure.account.oauth2.client.secret",dbutils.secrets.get(scope="",key=""))
spark.conf.set("fs.azure.account.oauth2.client.endpoint","https://login.microsoftonline.com//oauth2/token")

We need a feature in the tool to create an inventory of all service principals and direct files/mount points that are currently being used in the workspace along with the objects in teh spark session

[qziyuan]

I believe this is issue is not completed, since we are not crawling SPN from Spark session level settings in user codes

[nfx]

@qziyuan we can start working on it once we complete the table migration critical path

[nfx]

this is not in our critical path, as we can crawl for SPN usage directly through Azure APIs - https://github.com/databrickslabs/ucx/blob/main/src/databricks/labs/ucx/azure/access.py#L23-L31

[JCZuurmond]

this is not in our critical path, as we can crawl for SPN usage directly through Azure APIs - https://github.com/databrickslabs/ucx/blob/main/src/databricks/labs/ucx/azure/access.py#L23-L31

@dipankarkush-db : Could you confirm that ucx already covers your question? If the service principal is used through Spark settings, we find it due its permissions on storage accounts. We might not find its credentials, but we recommend to use access connectors instead.

If it covers your question, then we need to rewrite this issue to remove these spark configuration settings during code migrations as the authentication is handled by UC.

[nfx]

Closing in favour of #2021