Find Usage of DBFS Mounts in the Workspace and Repo

[dipankarkush-db]

Historically, customers have used mount points to connect their Databricks workspaces to their chosen cloud object storage, and many chose to utilize file paths as opposed to tables for accessing their data. For these long-time customers, the migration process is extremely manual, slow, and disruptive. This drastically decreases the chances of legacy customers upgrading their environment to Unity Catalog, which over time will lead to increased customer churn in favor of other platforms which do not require such a manual refactoring. Until now, we have not been able to provide an easier path forward for these customers.

[zpappa]

Functional Requirements

• Crawler must find usages of mount paths in Python/SQL in SQL and Spark Commands
• Crawler must find usages of mount paths in Python/SQL in Python dbutils.fs commands
• Crawler must find usages of mount paths in Python/SQL in shell commands
• Crawler must find usages of mount paths in Python/SQL in FS commands
• Crawler must inventory mount paths and where they are used, notebook, query, cell, line

Persisted Table

mount_path_usages

Field	Comment
mount_path_to_file	the entire mount path as used in code, i.e. /mnt/mount/myfile.txt
mount_path	the mount path, i.e. /mnt/mount
path_to_file	path to the notebook in the workspace or relative to the root of the repo, empty if a query
git_repo	git repo that is used
query_id	id of the query if a query, empty if a notebook
notebook_command_type	python, sql, sh, fs, etc
notebook_command_offset	the offset of the command if a notebook
line_number	the line number (not offset) of the usage in the query or file, or in the notebook cell

Implementation Details

• We have an existing implementation that works in uc-job-upgrade that needs to be ported in

Considerations

• Provide some way to scan only certain branches in git

[nfx]

Duplicate of #1133